Voidly incident timeline reconstruction: building the canonical event sequence from distributed probe measurements

The timeline reconstruction problem

When Voidly's probe network detects a censorship incident, the raw measurements do not arrive in chronological order. A probe in Tehran running a test cycle at 14:32 UTC fires first. A probe in Isfahan catches the block at 14:47 UTC. The following morning, CensoredPlanet's nightly batch export arrives — and it shows blocking beginning at 14:15 UTC, seventeen minutes before Voidly's earliest probe saw anything.

The canonical start_time for this incident must be revised backward to 14:15 UTC when the CensoredPlanet data arrives. Until that revision, every consumer of the incident record — journalists, researchers, automated alerting systems — is working with an underestimated duration. Across the 2025 incident corpus, this gap between first Voidly probe detection and the true incident start time has a median of 23 minutes. Maximum observed retroactive adjustment: 4.2 hours.

The challenge is not just about timestamps. Measurements arrive asynchronously because probes run on independent cycles, CensoredPlanet publishes as daily batches rather than a real-time stream, and network partitions can delay probe uploads by minutes or hours. A naive approach that takes the first-arriving probe's timestamp as start_timesystematically underestimates incident duration and misrepresents the incident sequence to downstream systems. Correct timeline reconstruction requires temporal alignment across time zones, confidence weighting by evidence quality, and a principled model for retroactive revision that preserves audit history.

The IncidentEvent sourcing model

Every state transition in an incident's lifecycle is recorded as an immutable event. Events are never deleted or modified — when new evidence changes our understanding of an incident, we insert a new event that supersedes the old one, with a revision_of pointer linking the revision to the event it replaces. This gives us a complete audit log of how Voidly's understanding of each incident evolved as evidence arrived.

The event type enumeration covers the full incident lifecycle:

type IncidentEventType =
  | 'FIRST_DETECTED'
  | 'PROBE_CONFIRMED'    // >=3 independent probes corroborate
  | 'ESCALATING'         // blocking rate crosses 60% threshold
  | 'PEAK'               // blocking rate >90% for >=2 consecutive windows
  | 'DEESCALATING'       // blocking rate drops below 70%
  | 'RESOLVED'           // all probes report clean for threshold period
  | 'RESOLUTION_REVISED' // late-arriving data extended or changed resolution
  | 'RETROACTIVE_START'  // start_time revised backward from batch data

interface IncidentEvent {
  incident_id: string;
  event_type: IncidentEventType;
  occurred_at: string;         // UTC ISO8601
  probe_count: number;
  blocking_rate: number;       // 0.0-1.0
  evidence_sources: string[];  // ['voidly_probes', 'ooni', 'censored_planet']
  confidence: number;          // 0.0-1.0
  revision_of?: string;        // event_id this supersedes (for RETROACTIVE_START)
}

The evidence_sources array records which measurement infrastructures contributed to the event. A PROBE_CONFIRMED event sourced only from Voidly carries ['voidly_probes']; a RETROACTIVE_STARTevent originating from the nightly CensoredPlanet batch carries ['censored_planet']. The confidence field is computed at event insertion time using the formula described in the confidence weighting section below.

The ESCALATING and DEESCALATING transitions are derived from the blocking rate measured across all active probes for the incident at a given time window. Blocking rate is computed as the fraction of probe measurements in a 15-minute window that classify as anomalous — not the fraction of all probes in the country, which would be confounded by probe coverage. A country with 3 probes all reporting blocking has a blocking rate of 1.0; a country with 30 probes where 18 report blocking has a blocking rate of 0.6.

The incident_timeline_events TimescaleDB hypertable

Timeline events are stored in a TimescaleDB hypertable partitioned by occurred_at. TimescaleDB's time-based partitioning ensures that queries over recent events — the most common access pattern — hit only the current chunk rather than scanning the full table. The secondary index on (incident_id, occurred_at DESC) makes per-incident timeline retrieval fast regardless of where in historical time the incident occurred.

CREATE TABLE incident_timeline_events (
  incident_id      TEXT NOT NULL,
  event_id         UUID DEFAULT gen_random_uuid(),
  event_type       TEXT NOT NULL,
  occurred_at      TIMESTAMPTZ NOT NULL,
  probe_count      INTEGER NOT NULL DEFAULT 0,
  blocking_rate    REAL NOT NULL,
  evidence_sources TEXT[] NOT NULL,
  confidence       REAL NOT NULL,
  revision_of      UUID,
  recorded_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
SELECT create_hypertable('incident_timeline_events', 'occurred_at');
CREATE INDEX ON incident_timeline_events (incident_id, occurred_at DESC);

The distinction between occurred_at and recorded_at is load-bearing. occurred_at is when the event actually happened — the timestamp carried by the probe measurement or the CensoredPlanet batch record. recorded_at is when the row was inserted into the database, which may be hours or days later for batch-sourced events. The hypertable is partitioned by occurred_at rather than recorded_at because incident timeline queries care about event order in incident time, not in database-insertion time. A RETROACTIVE_START event inserted today with an occurred_at of two days ago lands in the correct historical chunk.

No foreign key constraint links incident_id to the incidents table. TimescaleDB does not support foreign keys on hypertable partition columns, and the cross-table reference integrity is enforced at the application layer instead. Referential integrity violations are caught by a nightly consistency check that scans for timeline event incident_id values with no matching row in the incidents table.

Temporal alignment across time zones

Each probe measurement records probe_local_offset_secs — the probe's UTC offset at the time of measurement, in seconds. This field is stored at measurement time by the probe itself, not derived afterward from geographic data. The distinction matters for probes in jurisdictions that observe DST: a probe in a European country that crosses a DST boundary during a long-running incident would produce ambiguous timestamps if the offset were derived retroactively from location data. By storing the offset at measurement time, each timestamp carries its own unambiguous conversion factor.

Iran is a representative case. Iran operates on IRST (Iran Standard Time, UTC+3:30) year-round — the country does not observe DST. Probes in Tehran report probe_local_offset_secs = 12600 (3.5 hours × 3600 seconds/hour) regardless of season. The conversion from probe local time to UTC is straightforward:

from datetime import datetime, timezone, timedelta

def normalize_to_utc(local_ts: str, probe_local_offset_secs: int) -> datetime:
    local_dt = datetime.fromisoformat(local_ts)
    utc_offset = timedelta(seconds=probe_local_offset_secs)
    return local_dt.replace(tzinfo=timezone(utc_offset)).astimezone(timezone.utc)

The function handles both naive and timezone-aware ISO 8601 strings. datetime.fromisoformat returns a naive datetime for strings without a timezone suffix, and replace(tzinfo=...) attaches the probe-reported offset without shifting the time value. The subsequent astimezone(timezone.utc)then converts to UTC. If the input string already carries a timezone suffix, the probe offset is applied as the authoritative source, overriding any embedded offset — because probe-reported offsets have been validated against NTP-synchronized clocks during commissioning, whereas embedded offsets in uploaded measurement payloads sometimes reflect incorrect system clock configuration on the probe device.

Timeline reconstruction normalizes all probe timestamps to UTC before ordering. A measurement from Isfahan with a local timestamp of 18:02 IRST and probe_local_offset_secs = 12600 normalizes to 14:32 UTC — which may precede or follow a measurement from a probe in a different time zone whose local clock read an earlier hour. Ordering by raw local timestamps without normalization produces spurious sequencing errors that misrepresent which probe detected the block first.

Confidence weighting

An event's confidence score reflects how many independent probes and how many distinct ASNs have contributed evidence. Single-probe detections carry low confidence because a single probe may have a misconfigured resolver, an unusual network path, or a transient local anomaly unrelated to censorship. Evidence from multiple probes across multiple ASNs is much stronger.

The confidence formula:

confidence = min(1.0, (probe_count / 3.0) * 0.7 + (asn_count / 2.0) * 0.3)

The formula weights probe count at 70% and ASN count at 30%. Three probes in a single ASN yield a confidence of 0.7 — substantial, but not CONFIRMED, because all three share the same network path and the same DNS resolver. Two probes in two distinct ASNs yield 0.467 + 0.3 = 0.767 — CONFIRMED — because independent ASNs rule out resolver-level or path-specific artifacts.

The confidence tiers:

• TENTATIVE: confidence < 0.4. Typically a single probe with no cross-ASN corroboration. Surfaced in the API but marked with a warning; not included in the default journalist-facing dashboard.
• PROBABLE: confidence ≥ 0.4 and < 0.7. Multiple probes or a cross-ASN pair, but not yet meeting the full three-probe / two-ASN threshold. Surfaced in the dashboard with a confidence indicator.
• CONFIRMED: confidence ≥ 0.7. Three or more independent probes across two or more distinct ASNs. The standard threshold for including an event in public incident timelines and triggering journalist alerts.

CensoredPlanet batch evidence, when it arrives, can push an event from TENTATIVE or PROBABLE to CONFIRMED without additional Voidly probes. The formula treats CP vantage points as additional probes for the purpose of the numerator, up to a cap of 3 (since CP coverage in any single country is not independent in the ASN sense — CP probes may share transit paths). The cap prevents CP data from dominating the confidence score in ways that would mask Voidly probe gaps.

Retroactive revision from batch data

CensoredPlanet publishes nightly exports covering measurements from 50,000+ vantage points across residential, educational, and commercial networks worldwide. These exports arrive the morning after measurement, which means that for any given incident, CP data provides a retrospective view with higher geographic coverage than Voidly's real-time probe stream. When CP data shows that blocking started earlier than Voidly's first probe detection, the system executes a four-step revision:

Step 1: Insert a RETROACTIVE_START event into incident_timeline_events with occurred_at set to the CP-derived earlier start time. Set revision_of to the event_id of the original FIRST_DETECTED event. The original event remains in the table — it is never deleted — and the revision_of pointer allows the UI to display both events, with the revised one highlighted and the original dimmed for audit purposes.

Step 2: Update the start_time field on the incident record in the incidents table to the CP-derived timestamp. This is the only mutable field that changes — all other incident fields remain stable. The update also sets a start_time_revised boolean and a start_time_revision_source enum value (in this case, censored_planet).

Step 3: Log the revision in the incident_revisions audit table with the old value, new value, revision source, and the batch export filename that contained the evidence.

Step 4: Fire a REVISION webhook to all active subscribers with the delta payload:

{
  "incident_id": "inc_IR_20251217_a3f8c91b",
  "field": "start_time",
  "old_value": "2025-12-17T14:32:00Z",
  "new_value": "2025-12-17T14:15:00Z",
  "revision_source": "censored_planet"
}

Subscribers — primarily newsroom alerting systems and research pipeline consumers — receive the webhook within 90 seconds of the batch ingestion completing. The webhook payload is intentionally minimal: it carries only the changed field and its old and new values, not the full incident record, to minimize payload size for high-frequency revision events. Consumers that need the full updated record can fetch GET /v1/incidents/{incident_id} in response to the webhook.

Median retroactive adjustment in 2025: −23 minutes (incidents started earlier than Voidly first detected). Maximum observed: −4.2 hours, in a case where Voidly had no probe coverage in the affected country during the first hours of an incident and CP data provided the only early measurement window.

Timeline reconstruction query

The canonical timeline for an incident is assembled by querying the hypertable ordered by occurred_at ASC, recorded_at ASC. The secondary sort on recorded_at breaks ties when two events share the same occurred_at— which happens when a RETROACTIVE_START event and the original FIRST_DETECTED event share the CP-derived timestamp (if Voidly happened to first detect the incident at the same time CP shows it starting):

SELECT
  event_type,
  occurred_at,
  blocking_rate,
  evidence_sources,
  confidence,
  revision_of IS NOT NULL AS is_revision
FROM incident_timeline_events
WHERE incident_id = $1
ORDER BY occurred_at ASC, recorded_at ASC;

When is_revision is true, the UI renders the event in a highlighted revision style — typically a yellow left border and a “revised” badge. The original event that was superseded remains in the result set; the UI renders it dimmed with a strikethrough on the timestamp, and hovering shows the revision pointer. This dual display is important for trust: readers can see both the original detection timestamp and the CP-corrected timestamp side by side, with the revision source explicitly labeled. Hiding the original event would make the timeline appear to have always known the correct start time, which would obscure the inherent latency of distributed measurement.

Duration statistics after timeline reconstruction

After applying retroactive revision, the 2025 incident duration distribution by type shows heavily right-skewed tails driven by court-ordered long-running blocks:

The heavy right tail for DNS injection and HTTP block page incidents reflects court-ordered blocks in Turkey, Russia, and Indonesia that remain legally in force indefinitely — these are not operational anomalies but durable censorship policy. A DNS injection that a court ordered in 2022 targeting a news outlet appears in the p99 as a still-open incident in 2025. BGP blackholes resolve much faster because they are operationally expensive to maintain: routing infrastructure requires human action to sustain a prefix withdrawal, whereas a DNS injection can be left in place at a resolver level indefinitely with no ongoing cost.

Duration statistics are computed on the post-reconstruction timeline, meaning retroactive start time adjustments are incorporated before computing duration_hours = window_end - start_time. Without retroactive revision, the median BGP blackhole duration is underestimated by approximately 8 minutes; the median HTTP block page duration is underestimated by approximately 31 minutes, reflecting the longer gap between incident start and first probe detection for slow-starting HTTP blocks that a probe may cycle past before catching.

The timeline REST API endpoint

The timeline endpoint returns the full ordered event sequence for an incident, including confidence, evidence sources, and revision metadata. The ?since= parameter supports polling: consumers can request only events that occurred after a given ISO 8601 timestamp, enabling efficient incremental updates without re-fetching the full timeline on each poll.

A four-event example response for a confirmed Iran incident:

GET /v1/incidents/inc_IR_20251217_a3f8c91b/timeline

HTTP/1.1 200 OK
Content-Type: application/json

{
  "incident_id": "inc_IR_20251217_a3f8c91b",
  "events": [
    {
      "event_id": "e1a2b3c4-0001-0000-0000-000000000001",
      "event_type": "FIRST_DETECTED",
      "occurred_at": "2025-12-17T14:32:00Z",
      "probe_count": 1,
      "blocking_rate": 1.0,
      "evidence_sources": ["voidly_probes"],
      "confidence": 0.23,
      "revision_of": null,
      "is_superseded": true
    },
    {
      "event_id": "e1a2b3c4-0002-0000-0000-000000000002",
      "event_type": "PROBE_CONFIRMED",
      "occurred_at": "2025-12-17T14:47:00Z",
      "probe_count": 3,
      "blocking_rate": 0.94,
      "evidence_sources": ["voidly_probes"],
      "confidence": 0.84,
      "revision_of": null,
      "is_superseded": false
    },
    {
      "event_id": "e1a2b3c4-0003-0000-0000-000000000003",
      "event_type": "ESCALATING",
      "occurred_at": "2025-12-17T15:02:00Z",
      "probe_count": 5,
      "blocking_rate": 0.87,
      "evidence_sources": ["voidly_probes", "ooni"],
      "confidence": 0.96,
      "revision_of": null,
      "is_superseded": false
    },
    {
      "event_id": "e1a2b3c4-0004-0000-0000-000000000004",
      "event_type": "RETROACTIVE_START",
      "occurred_at": "2025-12-17T14:15:00Z",
      "probe_count": 12,
      "blocking_rate": 0.91,
      "evidence_sources": ["censored_planet"],
      "confidence": 0.78,
      "revision_of": "e1a2b3c4-0001-0000-0000-000000000001",
      "is_superseded": false
    }
  ],
  "canonical_start_time": "2025-12-17T14:15:00Z",
  "timeline_status": "ACTIVE"
}

The is_superseded flag on the original FIRST_DETECTED event signals to consumers that this event has been revised. The canonical_start_timefield in the response envelope always reflects the authoritative start time after all revisions have been applied — consumers that want a single authoritative timestamp without parsing the full event sequence can use this field directly.

The ?since= parameter filters by occurred_at, not recorded_at. This means a retroactive revision event with an occurred_at earlier than the polling window will not appear in incremental fetches — it will only appear in a full timeline fetch. Consumers that need to detect retroactive revisions should also subscribe to the REVISION webhook, which fires immediately when a revision is processed regardless of the event's occurred_at.

Integration with incident clustering

Timeline reconstruction feeds back into the clustering system in one significant edge case. When a RETROACTIVE_START event moves an incident's start_time backward by more than 6 hours, the clustering engine triggers a re-clustering pass for that incident's four-tuple key — the same (country_code, domain, interference_type, probe_type_group) key used during initial clustering.

The re-clustering pass checks whether the revised start time now overlaps with a previously-resolved incident for the same key. If an incident that was resolved at 13:50 UTC is followed by a new incident whose start time has just been revised to 13:45 UTC, the two incidents overlap — and what was recorded as two separate incidents may actually be one continuous event, with the gap between them being a measurement artifact rather than a genuine block-and-resume sequence.

When the re-clustering pass finds an overlap, it does not automatically merge the incidents. Instead, it creates a CLUSTERING_REVIEW flag on both incident records and queues them for the nightly manual review pipeline. Automated merges are deliberately avoided here because the decision to merge two incidents affects every downstream record that carries either incident_id — including published dataset exports that may already be cited in research papers. A merge that turns out to be incorrect is much harder to reverse than a flag that prompted a human review. The re-clustering threshold of 6 hours matches the standard gap rule: a retroactive adjustment smaller than 6 hours cannot create an overlap with a previously-resolved incident, because the gap rule already required a 6-hour clean window before closing the earlier incident.

In 2025, the re-clustering trigger fired on 14 incidents, of which 11 were confirmed merges after review and 3 were confirmed to be genuinely distinct incidents that happened to overlap in the revised timeline window. The 3 non-merges were cases where the earlier incident had resolved via a different mechanism (BGP re-announcement) while the later incident reflected a new DNS-layer block applied at roughly the same time — a pattern consistent with a blocking operator removing one control mechanism and immediately applying another.

For how incidents are first created — the four-tuple clustering key and 6-hour gap rule: Incident clustering and deduplication: how Voidly avoids counting the same censorship event twice →

For how Voidly determines a censorship incident has ended — per-type resolution thresholds: Voidly incident resolution: how we know when a censorship event ends →

For how cross-source corroboration from OONI and CensoredPlanet affects confidence: Cross-source censorship verification: reconciling OONI, CensoredPlanet, and IODA →

For the real-time event pipeline that processes measurements into alerts within 8 minutes: Voidly's real-time event pipeline: from measurement anomaly to journalist alert in under 8 minutes →