Voidly probe run lifecycle: from scheduled task to classifier input

The previous article in this series covered how a Voidly probe maintains its connection to the ingest endpoint — QUIC on port 443, Ed25519 batch authentication, domain fronting for blocked environments, and the SQLite buffer that prevents measurement loss during disconnections. That article stopped at the transport layer. This one picks up at the other end: what happens inside a single probe run, from the moment the scheduler dispatches a task to the moment a signed ProbeResult enters the upload queue.

A single probe run covers one domain and four protocol layers: DNS, TCP, TLS, and HTTP. Each phase captures structured data, flags anomalies it can detect locally, and passes its result forward to the next phase. After all four phases complete, the probe assembles a ProbeResult, signs it with the device key, and hands it to the upload worker. The whole sequence runs in under 700ms at the p50. This document walks through every step.

Scheduling trigger and the MeasurementTask

The measurement scheduler dispatches tasks over an in-process tokio mpsc channel with a capacity of 256. The probe's measurement loop reads one task at a time and executes it sequentially. Parallelism happens inside each task (DNS queries run concurrently via tokio::join!; the HTTP control fetch runs in parallel with the ISP fetch) but tasks themselves are not interleaved. Keeping tasks sequential avoids ambiguity when the same domain is measured by two overlapping runs within the same probe.

Each task carries everything the measurement loop needs:

pub struct MeasurementTask {
    pub domain:                   String,
    pub protocols:                Vec<Protocol>,  // [DNS, TCP, TLS, HTTP]
    pub priority:                 u8,             // 0 = lowest, 255 = urgent
    pub jitter_ms:                u16,            // ±15% random delay applied at task start
    pub scheduled_window:         DateTime<Utc>,
    pub expected_control_resolver: IpAddr,        // 8.8.8.8 / 1.1.1.1 / 9.9.9.9
}

The jitter_ms field encodes a random delay, drawn at schedule time, that the probe applies before starting the measurement. Jitter serves two purposes: it prevents probes in the same country from querying the same domain simultaneously (which would create a detectable traffic burst), and it makes periodic probe traffic harder to fingerprint by a network observer looking for regular measurement intervals. The jitter range is ±15% of the base measurement interval, capped at 500ms in practice.

// Measurement loop — reads tasks one at a time
async fn measurement_loop(
    mut rx: tokio::sync::mpsc::Receiver<MeasurementTask>,
    upload_tx: flume::Sender<ProbeResult>,
) {
    while let Some(task) = rx.recv().await {
        // Apply scheduling jitter before starting
        if task.jitter_ms > 0 {
            tokio::time::sleep(Duration::from_millis(task.jitter_ms as u64)).await;
        }

        match run_measurement(&task).await {
            Ok(result) => {
                let _ = upload_tx.send_async(result).await;
            }
            Err(e) => {
                tracing::warn!(domain = %task.domain, "measurement failed: {e}");
                // Task is silently dropped — counted in skip metrics
            }
        }
    }
}

Phase 1: DNS measurement

DNS is the first measurement phase and the most commonly affected by censorship. The probe queries the domain against two resolvers simultaneously using tokio::join!: the ISP-assigned resolver (obtained via DHCP) and a control resolver from the task's expected_control_resolver field (typically 8.8.8.8, 1.1.1.1, or 9.9.9.9, rotated across the fleet to avoid any single control resolver becoming a fingerprint). Both queries use standard UDP port 53 with a 3-second timeout.

pub struct DnsResult {
    pub resolver:          IpAddr,           // ISP resolver (DHCP-assigned)
    pub control_resolver:  IpAddr,           // 8.8.8.8 / 1.1.1.1 / 9.9.9.9
    pub isp_response:      DnsResponse,
    pub control_response:  DnsResponse,
    pub isp_rtt_ms:        f32,
    pub control_rtt_ms:    f32,
    pub anomaly:           Option<DnsAnomaly>,  // NXDOMAIN_MISMATCH | IP_MISMATCH | TIMEOUT
}

Running both queries concurrently via tokio::join! means the DNS phase duration is determined by the slower of the two resolvers, not their sum. At p50: ISP query takes 18ms, control query takes 12ms, total DNS phase is 22ms. At p99 the ISP resolver occasionally stalls — congested or deliberately delaying queries — driving the phase to 180ms.

The probe classifies DNS anomalies before proceeding to TCP. Three conditions set a DnsAnomaly:

• NXDOMAIN_MISMATCH: the ISP resolver returns NXDOMAIN for a domain that the control resolver resolves successfully. This is the canonical DNS blocking pattern — the ISP is lying about whether the domain exists.
• IP_MISMATCH: both resolvers return A/AAAA records, but the IP sets are disjoint (excluding CDN variation). This often indicates DNS hijacking to a block page server.
• TIMEOUT: the ISP resolver does not respond within 3 seconds, but the control resolver does. Selective timeout is a common censorship technique in networks where returning NXDOMAIN would be too obvious.

DNS anomaly detection here is the probe's own local judgment. It is not the final classification — the downstream classifier has access to the full 90-day per-domain per-country baseline and applies more sophisticated heuristics. The probe's local DnsAnomaly field is an input feature for the classifier, not a verdict.

Phase 2: TCP connect

TCP measurement always uses the IP returned by the ISP resolver, not the control resolver's IP. This is intentional: we are measuring the user's actual network path, not a hypothetical path through an unblocked IP. The comparison for anomaly detection is against a 90-day per-domain per-country TCP success rate baseline stored server-side.

pub struct TcpResult {
    pub target_ip:       IpAddr,
    pub port:            u16,
    pub connect_success: bool,
    pub rtt_ms:          Option<f32>,
    pub rst_injected:    bool,            // true if RST within 15ms of SYN
    pub rst_source:      Option<IpAddr>,  // IP that sent RST (may differ from target)
    pub error:           Option<TcpError>,
}

The RST injection detection deserves explanation. A common middlebox technique is to intercept the TCP SYN, allow it to reach the server, but inject a TCP RST packet toward the client before the server's SYN-ACK arrives. From the client's perspective the connection is refused, but from the server's perspective it succeeded. The signature is timing: an RST arriving within 15ms of the SYN almost certainly originates from an in-path device rather than the destination server (which would need at least one RTT to respond). When rst_injected is true, the probe also records rst_source — the IP address that sent the RST packet, which often belongs to the ISP's Deep Packet Inspection infrastructure rather than the target server.

TCP phase p50 is 28ms. At p99 it reaches 320ms, dominated by connection timeouts in countries where TCP connections to blocked destinations are dropped silently (no RST, no ICMP unreachable — the SYN just disappears).

Phase 3: TLS measurement

TLS measurement detects two distinct blocking techniques: SNI-based blocking and TLS MITM (certificate substitution). To distinguish them, the probe performs two TLS handshake attempts — one with the real domain as the SNI, and one with a blank SNI.

pub struct TlsResult {
    pub sni:                  String,
    pub cert_chain:           Vec<DerCertificate>,
    pub alert_code:           Option<u8>,          // TLS alert if handshake fails
    pub alert_rtt_ms:         Option<f32>,         // time from ClientHello to alert
    pub is_mitm:              bool,                // SPKI fingerprint mismatch vs. control
    pub control_cert_fp:      [u8; 32],            // SHA-256 of control SPKI
    pub measured_cert_fp:     Option<[u8; 32]>,
    pub negotiated_version:   Option<TlsVersion>,
}

The dual-SNI probing pattern works as follows: if the handshake with sni=domain fails with a TLS alert but the handshake with a blank SNI (connecting to the same IP) succeeds, the blocking is SNI-based — the network device is inspecting the ClientHello and terminating connections that carry a blocked hostname. This is the TSPU pattern documented in Russian network blocking research: the Transparent Proxy Unit sees the SNI in the ClientHello in plaintext and issues a TCP RST or TLS alert before the handshake completes.

An alert arriving within 30ms of the ClientHello is a strong indicator that the alert was generated by an in-path device rather than the destination server. A real server responding with a TLS alert (for instance, an unrecognized SNI) would need at least one round-trip. The alert_rtt_ms field captures this timing so the classifier can incorporate it as a feature.

MITM detection compares the SPKI SHA-256 fingerprint of the certificate received from the ISP's path against the control fingerprint obtained in the same measurement. If they differ — and the ISP-path certificate was issued by a CA not in the standard root store, or by a CA known to be operated by a state actor — is_mitm is set true. The TLS phase p50 is 180ms (including the TCP connection); the dual-SNI approach adds roughly 90ms in blocked cases.

Phase 4: HTTP measurement

The HTTP phase is the most information-rich and the most expensive. The probe fetches the URL over HTTP (not HTTPS — to observe any redirect or block page without TLS interference masking the content) and concurrently fetches the same URL from a Cloudflare Worker operating as a control vantage in a neutral region.

pub struct HttpResult {
    pub url:                   String,
    pub status_code:           Option<u16>,
    pub response_headers:      Vec<(String, String)>,
    pub body_length:           usize,
    pub body_simhash:          u64,            // 64-bit SimHash for blockpage matching
    pub ttfb_ms:               f32,            // time to first byte
    pub control_body_simhash:  u64,            // for comparison
    pub blockpage_match:       Option<BlockpageMatch>,  // from 2300-entry library
    pub error:                 Option<HttpError>,
}

Body comparison uses three signals in combination:

• body_length_ratio: ISP body length divided by control body length. A ratio below 0.2 (the ISP returned far less content than the control) or above 5 (the ISP returned far more, suggesting an injected page) is flagged as suspicious.
• SimHash distance: the 64-bit SimHash is computed over the response body tokens. Two hashes within 8 bits of Hamming distance are considered identical content; hashes more than 48 bits apart are considered entirely different documents. The SimHash tolerates minor dynamic content differences (timestamps, session tokens) that would cause a byte-for-byte comparison to fail.
• status_code mismatch: a 200 from the control and a 302 or 403 from the ISP path is a clear signal. Block pages sometimes return 200 with a redirect body — the SimHash comparison catches these where the status code does not.

The blockpage_match field reflects a lookup against a 2,300-entry library of known block page fingerprints. Each library entry is a SimHash cluster centroid from known block pages collected across 90+ countries. A match here is high-confidence evidence of active content injection — the ISP is serving a government-mandated block page rather than simply dropping the connection.

HTTP phase p50 is 380ms, including the concurrent control fetch. At p99 it reaches 1,800ms, driven by slow ISP-side responses in high-latency environments and occasional control vantage timeouts.

ProbeResult assembly and signing

After all four phases complete, the probe assembles a ProbeResult and signs it:

pub struct ProbeResult {
    pub probe_id:      ProbeId,
    pub domain:        String,
    pub measured_at:   DateTime<Utc>,
    pub dns:           DnsResult,
    pub tcp:           TcpResult,
    pub tls:           TlsResult,
    pub http:          HttpResult,
    pub quality_flags: QualityFlags,
    pub signature:     Ed25519Signature,  // over hash of all fields
}

The signature is computed over a canonical hash of all result fields using the probe's Ed25519 device key. This is the same key embedded in the device certificate issued during probe commissioning. The ingest worker at the Cloudflare Worker layer verifies this signature before accepting the batch — any tampering with the result fields in transit causes the signature check to fail and the batch to be rejected.

Quality flags are set by the probe itself before signing, based on conditions it can observe locally:

• control_unreachable: the control resolver or control HTTP vantage failed to respond during this measurement. Results with this flag are excluded downstream — without a working control we cannot distinguish censorship from local network failure.
• old_probe_version: the probe binary is older than version 2.5.0. Results are accepted but weighted lower in the classifier, since older probes lack some measurement fields.
• dedup_hit: the same domain was measured by this probe within the last 10 minutes. This can happen when an urgent injection from the scheduler overlaps with a regular scheduled run. The downstream deduplication logic retains only the most recent result in the 10-minute window.

fn compute_quality_flags(
    task: &MeasurementTask,
    dns: &DnsResult,
    http: &HttpResult,
    last_measured: &HashMap<String, DateTime<Utc>>,
    probe_version: &Version,
) -> QualityFlags {
    QualityFlags {
        control_unreachable: dns.control_response.is_err()
            || http.control_body_simhash == 0,
        old_probe_version: probe_version < &Version::parse("2.5.0").unwrap(),
        dedup_hit: last_measured
            .get(&task.domain)
            .map(|t| Utc::now() - t < chrono::Duration::minutes(10))
            .unwrap_or(false),
    }
}

Result batching and upload

Completed ProbeResult values are accumulated in memory. The upload worker flushes them under two conditions: when 50 results have accumulated, or when 5 minutes have elapsed since the last flush — whichever comes first. The 5-minute timer ensures that even low-activity probes upload regularly rather than waiting indefinitely.

Before upload the batch is serialized to protobuf and compressed with zstd at level 1. Level 1 is chosen for speed over compression ratio: result serialization must complete quickly so the upload does not delay the next measurement cycle. At level 1, a typical 50-result batch compresses from roughly 95KB to 28KB in under 2ms.

async fn flush_batch(
    results: Vec<ProbeResult>,
    transport: &ProbeTransport,
) -> Result<(), UploadError> {
    let batch = ProbeResultBatch {
        probe_id: PROBE_ID.clone(),
        results,
    };

    // Serialize to protobuf, then compress with zstd level 1
    let proto_bytes = batch.encode_to_vec();
    let compressed  = zstd::encode_all(proto_bytes.as_slice(), 1)?;

    // Upload via QUIC stream; exponential backoff on failure
    let mut attempts = 0u8;
    loop {
        match transport.upload_batch(&compressed).await {
            Ok(_) => return Ok(()),
            Err(e) if attempts < 3 => {
                let delay = Duration::from_secs(match attempts {
                    0 => 30,
                    1 => 300,
                    _ => 1200,  // 30s / 5m / 20m
                });
                tracing::warn!("upload attempt {}: {e}; retrying after {delay:?}", attempts + 1);
                tokio::time::sleep(delay).await;
                attempts += 1;
            }
            Err(e) => {
                tracing::error!("batch dropped after 3 failed uploads: {e}");
                return Err(UploadError::MaxRetriesExceeded);
            }
        }
    }
}

On upload failure the probe retries with exponential backoff: 30 seconds, 5 minutes, then 20 minutes. After three failures the batch is dropped and the loss is logged to the probe's telemetry stream. This is a deliberate tradeoff: holding failed batches indefinitely would grow memory without bound in environments where the ingest endpoint is persistently unreachable. The networking layer described in the previous article handles longer-duration disconnections with the SQLite buffer, but in-flight batches that fail after the buffer threshold are dropped.

Per-phase timing breakdown

Here is the complete timing picture for a single probe run, from scheduler dispatch to upload queue entry:

The p99 total of 3,200ms is dominated by the HTTP phase in high-latency environments. The 500ms p99 for the scheduler-to-task-start transition reflects the full jitter window for high-priority tasks dispatched with maximum jitter applied.

Error budget and effective throughput

Not every scheduled task produces a usable row in TimescaleDB. Losses occur at two distinct points in the pipeline:

At the probe itself, 7.8% of tasks are skipped before a measurement is even attempted. The primary reasons: dedup_hit (the scheduler injected an urgent re-measurement of the same domain within the 10-minute dedup window), control_unreachable (the control resolver failed before the DNS phase began, making the measurement uninformative), domain not resolving at the control (the domain itself may be down, not blocked), and jitter window overflow (a task's jitter delay pushed it past the next scheduler cycle boundary).

At the ingest layer, a further 3.2% of successfully uploaded measurements are dropped by the quality filter. The dominant reason here is control_unreachable persisting into the upload (the probe set the flag but uploaded anyway; the ingest filter enforces the exclusion). The remaining drops are probe-revoked measurements and late arrivals (measured_at more than 48 hours before upload, usually from probes that experienced extended disconnections).

Net effective throughput: approximately 89% of scheduled tasks produce a usable row in TimescaleDB. The 11% loss is acceptable given the measurement redundancy across probes — any given domain is scheduled on multiple probes in the same country, so a single skipped task does not create a measurement gap. The quality filter article covers the ingest-side drop logic in detail.

From ProbeResult to classifier input

Once a ProbeResult passes the quality filter and is inserted into TimescaleDB, it enters the classifier queue. The feature extraction layer transforms the four phase structs into the numeric feature vector that the anomaly classifier operates on: boolean flags for each anomaly type, RTT ratios (ISP vs. control), SimHash distances, TLS alert timing, RST injection indicators, and the historical baseline delta for TCP success rate.

The classifier assigns one of five interference classes (or clean) and a confidence score. That score, combined with cross-probe aggregation and external corroboration from OONI and IODA, drives the confidence tier system described in the real-time pipeline article — which is where this probe run's data finally surfaces as a Verified Incident or Corroborated event in the public dataset.

Voidly's probe networking layer: QUIC batching, Ed25519 authentication, and upload reliability →

Voidly's probe-to-dataset ingest pipeline: normalization, quality filtering, and TimescaleDB indexing →

Voidly's real-time event pipeline: from measurement anomaly to journalist alert in under 8 minutes →

The Voidly measurement scheduler: how we decide which domains to probe and when →