Technical writing
Voidly BGP data ingestion: parsing MRT dumps, detecting prefix withdrawals, and computing country outage scores
The previous article in this series covered Voidly's high-level design for BGP-based shutdown detection: why prefix withdrawals precede application-layer interference, how per-country baselines work conceptually, and why BGP silence and BGP withdrawal are not the same signal. This article goes one level deeper into the actual collection pipeline: which data sources feed it, how MRT dumps are fetched and parsed, how CountryBgpBaseline records are built and refreshed, how withdrawal events are classified by severity, and exactly how the resulting bgp_outage_score value ends up attached to probe measurements in the voidly_measurements table.
BGP data sources and collection cadence
Voidly draws from three independent BGP data sources. Each has a different latency profile and a different role in the pipeline.
RIPE NCC RIS is the primary source for both full routing table snapshots and incremental updates. The RIS network operates 26 route collector nodes (rrc00 through rrc26) distributed across major IXPs and transit exchange points globally. Full RIB snapshots are published every 8 hours as gzip-compressed MRT files, typically around 350 MB each. Incremental update files are published every 5 minutes at the top of each 5-minute mark, and run around 2 MB compressed. The collection loop fetches each update file via HTTP GET:
https://data.ris.ripe.net/rrc{00..26}/latest-bview.gz # full RIB snapshot
https://data.ris.ripe.net/rrc{00..26}/updates.{YYYYMMDD}.{HHmm}.gz # 5-min updatesRouteViews provides a second independent view of the global routing table. Full RIB dumps are available daily via HTTP fromarchive.routeviews.org/bgpdata/..., and 15-minute incremental update files are also available. RouteViews uses the MRT TABLE_DUMP_V2 format for full RIBs and BGP4MP for updates, the same as RIS. Having two independent route collector networks matters for false-positive suppression: a withdrawal seen at both RIPE NCC RIS and RouteViews has passed through completely separate BGP session infrastructure and is far less likely to be a collector artifact.
bgp.tools streaming API is a WebSocket feed of real-time BGP announcements and withdrawals, aggregated from multiple looking-glass and collector feeds. The feed carries roughly 50,000 events per hour globally during normal BGP activity. Voidly subscribes to this feed as a low-latency supplement during active shutdown events: when a withdrawal event is already in progress and confirmed, the bgp.tools stream provides sub-90-second confirmation of whether the event is deepening or recovering. Outside of confirmed events the stream is monitored but not used as a primary trigger, because its higher event volume also carries more noise.
MRT format and what it contains
MRT (Multi-Threaded Routing Toolkit) is the standardized binary format used by all major BGP route collectors to record routing table contents and BGP UPDATE messages. Two MRT message types are relevant to Voidly's pipeline:
Type 13 (TABLE_DUMP_V2) encodes a full RIB snapshot. Each entry contains a peer index (identifying which BGP session the route was learned from), the IP prefix (IPv4 or IPv6), and path attributes including AS_PATH, NEXT_HOP, ORIGIN, and COMMUNITIES. These are the full-RIB snapshot files, used to rebuild the per-country prefix count baselines.
Types 16 and 17 (BGP4MP) encode incremental BGP UPDATE messages. Each BGP UPDATE contains either WITHDRAWN_ROUTES (a list of prefixes being removed) or NLRI fields (new reachability announcements), or both. These are the 5-minute update files that feed the real-time withdrawal detection loop.
Voidly parses MRT files using bgpkit-parser, a Rust library with a Python binding that handles both TABLE_DUMP_V2 and BGP4MP formats, including compressed input via HTTP streaming. The parser normalizes both announcement and withdrawal events into a uniform BgpUpdate record:
from bgpkit_parser import BgpkitParser
def parse_mrt_updates(url: str) -> Iterator[BgpUpdate]:
"""Parse a live or archived MRT update file."""
parser = BgpkitParser(url)
for elem in parser:
yield BgpUpdate(
timestamp=elem.timestamp,
peer_asn=elem.peer_asn,
prefix=elem.prefix,
origin_as=extract_origin_as(elem.as_path),
update_type=UpdateType.ANNOUNCEMENT if elem.type == 'A' else UpdateType.WITHDRAWAL,
)The extract_origin_as function takes the rightmost AS in the AS_PATH attribute, which by BGP convention is the origin autonomous system — the AS that originally announced the prefix to the rest of the internet. This is the ASN we use for country mapping.
Origin ASN to country mapping
The core challenge in per-country prefix counting is correctly attributing each prefix to a country. Not every prefix can be attributed unambiguously: large transit providers route prefixes that span many countries, and some ASNs are registered in one jurisdiction but serve users in another.
Voidly builds the asn_country_map from two inputs: CAIDA's AS-Rank dataset for ISP classification and AS-to-organization mapping, and MaxMind GeoIP for country assignment of IP space. For each origin ASN, the map records the country that accounts for the largest fraction of that ASN's routed prefixes. A prefix is counted toward a country only when at least 60% of its origin ASN's routed prefixes geolocate to that country. ASNs below that threshold are classified as multi-country transit providers, and their prefixes are allocated proportionally across the countries they serve rather than assigned exclusively to one.
This proportional allocation matters for large transit ASNs that appear in the routing tables of many countries. An ASN like Cogent (AS174) routes prefixes in dozens of countries; assigning all its prefixes to the US would inflate the US baseline and deflate every other country's prefix count, making withdrawal detection noisier for countries that depend on transit through large US-headquartered carriers. The 60% threshold was chosen by backtesting against known shutdown events to minimize false positives while maintaining sensitivity for country-specific events.
Per-country baseline computation
The baseline for a country is the 90-day rolling distribution of its daily active prefix count. Each day, after the RIPE NCC full RIB dump is processed (at 02:00 UTC), the normalization pipeline counts active prefixes per country and updates theCountryBgpBaseline record:
@dataclass
class CountryBgpBaseline:
country_code: str
prefix_count_median: int # median over 90 days
prefix_count_p5: int # 5th percentile (lower bound)
prefix_count_p95: int # 95th percentile (upper bound)
last_updated: datetime
sample_count: int # number of daily snapshots
def update_baseline(cc: str, current_count: int, db: Session) -> None:
"""Called once per day after full RIB parse."""
baseline = db.query(CountryBgpBaseline).filter_by(country_code=cc).first()
# Append to rolling 90-day sample, recompute percentiles
...The 5th and 95th percentile bounds serve different roles. The p5 lower bound establishes the minimum expected prefix count for that country under normal conditions — a count below p5 is anomalous by definition, even before comparing against the median. The p95 upper bound is used to detect unusual announcement surges (route leaks, prefix hijacks) that could inflate the baseline if not handled carefully.
Not every country has sufficient BGP diversity for a reliable baseline. The current threshold is 100 distinct ASNs with active prefix announcements: 53 countries meet this threshold and receive individual baselines. The remaining 147 countries use a regional grouping fallback, where the baseline is computed across all countries in the same ITU region and normalized by each country's historical share of that regional prefix pool. Regional baselines are less sensitive but still catch large country-level events.
Withdrawal detection algorithm
The detection loop runs every 5 minutes, triggered by the arrival of a new RIPE NCC RIS update file. For each country, it computes the current active prefix count from the most recent update window and compares it against the 90-day median baseline. The withdrawal rate is the fraction of normally-advertised prefixes that are no longer being announced:
WITHDRAWAL_THRESHOLDS = {
BgpSeverity.LOW: 0.10, # 10% of prefixes withdrawn
BgpSeverity.MEDIUM: 0.25, # 25%
BgpSeverity.HIGH: 0.50, # 50%
BgpSeverity.CRITICAL: 0.80, # 80%
}
DETECTION_WINDOW_MINUTES = 15
def detect_withdrawal_event(
cc: str,
current_prefix_count: int,
baseline: CountryBgpBaseline,
window_start: datetime,
) -> Optional[BgpWithdrawalEvent]:
withdrawal_rate = 1 - (current_prefix_count / baseline.prefix_count_median)
severity = next(
(s for s, t in sorted(WITHDRAWAL_THRESHOLDS.items(), reverse=True)
if withdrawal_rate >= t),
None
)
if severity is None:
return None
return BgpWithdrawalEvent(
country_code=cc,
severity=severity,
withdrawal_rate=withdrawal_rate,
current_prefix_count=current_prefix_count,
baseline_prefix_count=baseline.prefix_count_median,
detected_at=datetime.utcnow(),
window_start=window_start,
)The severity thresholds were calibrated against historical shutdown events. LOW (10% withdrawal) covers events like partial ISP outages or regional infrastructure failures that warrant monitoring but not alerting. CRITICAL (80% withdrawal) covers events like the 2019 Ethiopia election-period shutdown, where the country's international BGP presence dropped by over 90% within a single 15-minute window. In practice, country-level government-ordered shutdowns tend to cluster at HIGH or CRITICAL: partial shutdowns (social media platforms only, for example) typically do not appear at the BGP level at all, because those platforms are hosted externally and their prefixes are not under the government's control.
BgpEvent record in TimescaleDB
When a withdrawal event is detected and passes false-positive checks (covered below), a BgpEvent record is written to the bgp_eventsTimescaleDB hypertable. The table is indexed on (country_code, started_at)for efficient range lookups by country and time window:
@dataclass
class BgpEvent:
event_id: uuid.UUID
country_code: str
severity: BgpSeverity # LOW/MEDIUM/HIGH/CRITICAL
withdrawal_rate: float # fraction 0-1
prefix_count_at_event: int
baseline_prefix_count: int
started_at: datetime
peak_withdrawal_at: Optional[datetime]
ended_at: Optional[datetime] # None if ongoing
estimated_affected_users: Optional[int] # from ITU broadband subscriber data
sources: List[str] # ['ripe_ris', 'routeviews', 'bgp_tools']The sources field records which BGP data sources observed the withdrawal. An event confirmed by all three sources carries higher evidentiary weight than one seen only in RIPE NCC RIS. The estimated_affected_usersfield is computed from ITU broadband and mobile subscriber data for the country, scaled by the withdrawal rate: a 60% withdrawal rate in a country with 40 million broadband subscribers produces an estimate of approximately 24 million affected users. These estimates carry significant uncertainty and are labeled as such in the public dataset.
Events remain open (ended_at IS NULL) until the prefix count recovers above the LOW threshold and stays there for at least two consecutive 5-minute windows. This prevents rapid oscillation from generating a flood of short-duration event records during flapping.
False positive mitigation
Not every prefix withdrawal indicates government-ordered censorship, and the false positive problem is non-trivial: the global BGP table sees constant churn from routine maintenance, peering changes, and infrastructure failures. Voidly applies five filters before writing a BgpEvent record:
Planned maintenance check. The RIPE Atlas maintenance calendar is fetched every 15 minutes. If a scheduled maintenance window overlaps the detection window for a country, the withdrawal event is held in a pending state and not surfaced until the maintenance window closes. If the withdrawal persists after the scheduled maintenance ends, it is then written as a BgpEvent.
Single-ASN filter. If fewer than 3 distinct ASNs contributed to the observed withdrawals, the event is classified as a probable routing incident rather than a government shutdown. A country-level shutdown requires the participation of all major transit providers in the country; a single AS withdrawing its prefixes is almost always a local routing problem, a BGP session reset, or a peering dispute.
Symmetric withdrawal detection. If multiple unrelated countries simultaneously show elevated withdrawal rates, the signal is treated as global BGP instability rather than country-specific action. The threshold for symmetric detection is 5 or more countries crossing the LOW threshold within the same 15-minute window. Major internet infrastructure events (route leaks, RPKI misconfiguration) can cause widespread simultaneous withdrawals that would otherwise trigger per-country alerts.
Short-duration filter. Withdrawals that resolve within 5 minutes are suppressed entirely. Transit path flapping, BGP session resets, and brief equipment failures typically produce sub-5-minute withdrawal events. Government-ordered shutdowns do not resolve in 5 minutes.
Confirmation requirement. A bgp_outage_scoreof 0.5 or above triggers a PROBABLE_SHUTDOWN flag in the probe measurement database, but the event is not promoted to VERIFIED status without corroboration from probe measurements. If Voidly's probes in the affected country also show elevated HTTP and DNS interference rates, the event moves to VERIFIED. BGP withdrawal alone is treated as a necessary but not sufficient condition for a verified shutdown determination.
Attaching bgp_outage_score to probe measurements
The bgp_outage_score field in voidly_measurements is computed at measurement write time via a SQL subquery against the bgp_eventstable. The score is a numeric translation of the event severity at the time of the measurement:
-- When writing a measurement row, join with bgp_events
SELECT
COALESCE(
(
SELECT CASE be.severity
WHEN 'CRITICAL' THEN 1.0
WHEN 'HIGH' THEN 0.8
WHEN 'MEDIUM' THEN 0.5
WHEN 'LOW' THEN 0.2
END
FROM bgp_events be
WHERE be.country_code = $1
AND be.started_at <= $2
AND (be.ended_at IS NULL OR be.ended_at >= $2 - INTERVAL '6 hours')
ORDER BY be.severity DESC
LIMIT 1
),
0.0
) AS bgp_outage_scoreThe 6-hour lookback on ended_at is intentional. BGP recovery — prefixes being re-announced — does not immediately translate to restored connectivity at the application layer. DNS poisoning, packet-level filtering, and CDN cache invalidation can persist for hours after routes are restored. A probe measurement taken 4 hours after a BGP event ended still occurred in a disrupted network environment, and excluding it from the outage context would misrepresent the measurement's provenance. The 6-hour window was chosen by examining historical recovery curves from known shutdown events; most application-layer interference resolves within 3–4 hours of BGP recovery, and 6 hours provides a conservative buffer without introducing false positive inflation for unrelated later measurements.
The ORDER BY be.severity DESC LIMIT 1 handles overlapping events: if a country has both a MEDIUM and a HIGH event overlapping the same measurement timestamp (possible during a rapidly-escalating shutdown), the measurement receives the higher severity score.
End-to-end detection latency
The pipeline is optimized for sub-two-minute latency from BGP UPDATE message to a written BgpEvent record. Each stage contributes:
Stage p50 ───────────────────────────────────────────────────────── RIPE NCC RIS update file published 0s (trigger) Download + decompress 2MB update file 30–60s Parse MRT updates (bgpkit-parser, async Rust) 8s Baseline comparison + withdrawal detection 200ms BgpEvent write to TimescaleDB 5ms ───────────────────────────────────────────────────────── BGP UPDATE → BgpEvent in database ~90s (p50)
The dominant cost is the download of the update file itself. RIPE NCC publishes files on a fixed 5-minute schedule, and the download begins immediately after the file is available. The bgpkit-parser MRT processing step runs in an async Rust task and overlaps with the download of the next file, so in steady state the pipeline is effectively I/O bound on the file fetch, not on parsing.
During active shutdown events, the bgp.tools WebSocket feed reduces the end-to-end latency to approximately 35 seconds (p50) by providing real-time event notifications without the 5-minute publish cadence of the update file schedule. The stream is activated automatically when a BgpEvent crosses the MEDIUM severity threshold and is disabled again when the event resolves.
For the high-level design of Voidly's BGP-based shutdown detection — why prefix withdrawals are the earliest signal and how BGP fits into the composite censorship score: BGP routing signals and internet shutdown detection: how Voidly uses IODA data →
For the raw probe byte-to-TimescaleDB pipeline that receives the bgp_outage_scorecomputed here: Voidly's probe-to-dataset ingest pipeline: normalization, quality filtering, and TimescaleDB indexing →
For the full field-by-field schema of bgp_outage_score and all other fields in the published measurement dataset: The Voidly measurement dataset: field-by-field schema reference →
For how bgp_outage_score combines with DNS, HTTP, and TLS interference probabilities to produce the per-country censorship index: Voidly's country-level censorship score: aggregating 2.2B probe measurements into the global index →