Writing · topic · 82 articles
The Voidly index: internet censorship measurement and shutdown detection, with companion datasets on surveillance, banned books, and information rights.
Information-rights posture has two axes — how open the government is (Right to Information) and how protected the citizen is (Data Protection). Joining the two new Voidly datasets on country (a rare exact key), this maps the two-by-two space and the real tension where privacy law is used to deny access and openness without protection exposes individuals.
Censorship and information control · Cybersecurity and privacy · Transparency and open data
Voidly is fourteen datasets on how accountability is suppressed and reclaimed: network censorship, banned books (Verboten), the surveillance industry (SpyLedger), ownership opacity (DarkRegister), foreign-held land (Foreign-Held U.S. Farmland), foreign money in universities (Section 117 Ledger), grid ownership (GridOwners), who runs ICE detention (the Detention Ledger), who signed up to enforce immigration law (the 287(g) Wave), every federal prison (the BOP Ledger), the sanctions authorities behind designations (Sanctions Programs), information rights (Right to Information and Data Protection), and the US organ system (OrganWatch). One shared source-cited, static, agent-first, privacy-careful method.
Censorship and information control · Transparency and open data
The public-access status of 46 national beneficial-ownership registers (EU, UK, US, and major offshore centres) after the 2022 CJEU ruling — only 7 remain fully public. A record of corporate-transparency rollback as state behavior, with zero personal data, plus the open CC0 GLEIF ownership graph captured as the preservable counterweight.
Censorship and information control · Transparency and open data
The public corporate identity and government-designation status of 20 marquee spyware and mass-surveillance vendors (NSO Group, Intellexa, Hikvision, Huawei and more) — every designation rebuilt from a primary US/EU source and precisely typed: export control, sanction, equipment-authorization, or investment restriction.
Censorship and information control · Sanctions and illicit finance · Transparency and open data
A data read of the Verboten index across 119 countries: political content is the world’s #1 stated reason for banning a book (9,813 titles), LGBTQ+ bans are ~95% American, and the 2020s already hold 9,411 newly banned titles — the two censorship regimes and the four-century arc behind them.
Censorship and information control · Transparency and open data
A structured, source-cited index of book censorship worldwide — 19,283 banned or restricted titles across 119 countries — built on the CC-BY banned-books.org Open Censorship Core and served as static JSON for AI agents to query: is this book banned in that country, and why?
Censorship and information control · Transparency and open data · Engineering and infrastructure
How Voidly deduplicates thousands of probe measurements into discrete censorship incidents: the four-tuple clustering key, the 6-hour gap rule, incident lifecycle from ANOMALY to RESOLVED, incident_id assignment, retroactive CensoredPlanet alignment, and edge cases including flapping blocks and BGP outages.
Censorship and information control · Engineering and infrastructure
How Voidly reconstructs the authoritative timeline of a censorship incident from asynchronous distributed probe measurements — IncidentEvent sourcing model, temporal alignment across time zones, confidence weighting requiring 3+ independent probes, retroactive revision from CensoredPlanet batch data, duration statistics, and the timeline REST API endpoint.
Censorship and information control · Engineering and infrastructure
How Voidly determines that a censorship incident has ended: per-type resolution thresholds (consecutive passing measurements with p_blocked < 0.3), the 12-hour RESOLVED_PENDING re-open window, FLAPPING state detection for rapidly alternating blocks, BGP-type auto-resolution, and cross-source confirmation requirements for VERIFIED incidents — with observed resolution time distributions (BGP 4.2h median, HTTP 12.1 days).
Censorship and information control · Engineering and infrastructure
How Voidly embeds ONNX Runtime inside an Apache Flink streaming job to score probe results for censorship anomalies at 50,000 events/sec with sub-100ms end-to-end latency: thread-local ONNX session management per task slot, Kafka partition alignment with (country_code, asn) keyBy, mini-batch coalescing for 50ms p99 inference, and the backpressure mechanism that keeps consumer lag under 2,400 messages even on election-day traffic spikes.
Censorship and information control · Engineering and infrastructure · Machine learning and OSINT
How Voidly gets from a probe anomaly to a published verified incident — and an alert in a journalist's inbox — in under 8 minutes: the event queue, real-time OONI and IODA API polling, confidence threshold crossing, the two-window alert-fatigue guard, and the nightly CensoredPlanet retroactive pass.
Censorship and information control · Engineering and infrastructure
What happens inside a single Voidly probe run: the measurement execution loop, DNS and TCP and TLS and HTTP data capture, result serialization and signing, and the upload path that delivers a signed ProbeResult to the ingest pipeline.
Censorship and information control · Engineering and infrastructure
How Voidly probes maintain connectivity and upload measurements from networks that actively block VPN protocols — QUIC/443 transport, domain fronting via CDN SNI fronting, TLS certificate pinning against MITM, local SQLite buffering (500 MB cap, 48h window), and metered-connection backoff.
Censorship and information control · Engineering and infrastructure
How Voidly probes preserve measurement data during upload failures — a 72-hour SQLite ring buffer with anomaly-safe eviction, LZ4 batch compression reducing median batch size from 47KB to 9KB, exponential backoff retry up to 4 hours, priority queue for anomalous measurements, chunked upload with per-chunk acknowledgment, and 0.003% measurement loss rate across 37 probes over 6 months.
Censorship and information control · Engineering and infrastructure
How the Voidly desktop probe works: Tauri 2 cross-platform app, Cloudflare boringtun WireGuard, tun-rs TUN device, X25519-Dalek on-device key generation, and operator anonymity as a design constraint.
Censorship and information control · Engineering and infrastructure
How the Voidly probe test runner orchestrates concurrent measurements inside the Tauri app: tokio Semaphore with 3 permits, MeasurementState machine (Pending → Running → Success/Error/Timeout), per-layer timeout budgets (DNS 3s, TCP 5s, TLS 8s, HTTP 15s, total 30s), Ed25519 measurement signing, mpsc upload queue with capacity 200, and why per-layer timeouts are themselves evidence of DNS-layer interference.
Censorship and information control · Engineering and infrastructure
A step-by-step breakdown of how each Voidly probe test works: DNS resolution, TCP handshake, TLS negotiation with certificate chain validation, HTTP request execution, response body fingerprinting, control comparison, and how every layer maps to interference types in the anomaly classifier.
Censorship and information control · Engineering and infrastructure
A deep dive into the TCP layer of Voidly's censorship detection: SYN-ACK timing, RST injection detection with a 15ms threshold, null-routing vs. RST as two distinct censorship mechanisms, the TcpResult struct, dual-IP probing to identify RST source, and how TCP evidence maps to the anomaly classifier's interference classes.
How Voidly uses a distributed control server network to distinguish genuine censorship from network errors, CDN split-horizon DNS, and misconfigured sites — DNS, TCP, TLS, and HTTP comparison methodology, and why a single control is not enough.
Censorship and information control · Engineering and infrastructure
A technical deep-dive on how Voidly detects bandwidth throttling — the hardest interference class to classify. Covers the TimingFeatures Rust struct, TTFB z-score computation against control measurements, body truncation and mid-transfer RST signals, the congestion vs. deliberate-throttling calibration problem, cross-probe corroboration scoring, and country patterns from Russia TSPU, Iran ARRS, India, and China.
Censorship and information control · Engineering and infrastructure
How Voidly monitors 37+ probe nodes: heartbeat system (60s cadence, separate transport), DEGRADED/OFFLINE state machine, measurement quality scoring, ASN coverage SLOs for 200 countries, flapping detection capping confidence at CORROBORATED, automated replacement from standby operator waitlist, and the classify_offline_cause() algorithm distinguishing probe failure from ISP-level censorship.
Censorship and information control · Engineering and infrastructure
How Voidly probes identify DNS injection and manipulation in censored networks — comparison against three control resolvers, four weighted detection signals (IP divergence, TTL anomaly, source IP divergence, response timing), per-country injection rates (China 94%, Iran 61%, Russia 12%), CAP_NET_RAW privilege handling, anycast false-positive calibration from 4.2% to 0.8%, and integration with the DnsTestResult confidence score.
Censorship and information control · Engineering and infrastructure
How Voidly avoids false positives from commercial geoblocking: HTTP 451 detection, streaming service block page fingerprints (tagged geoblock_commercial, not censorship), multi-country probe comparison (SINGLE_COUNTRY vs. MULTI_COUNTRY_SELECTIVE geographic patterns), CDN split-horizon detection via ASN group mapping, domain-level unavailability baselines, and the p_geoblock score that suppresses measurements above 0.70.
Censorship and information control · Engineering and infrastructure
How Voidly classifies every censorship measurement into one of 7 interference types — DnsInjection, DnsNxdomain, TcpRstInjection, TcpNullRouting, TlsMitm, HttpBlockPage, and Throttling — using a hierarchical decision tree from DNS through HTTP, with confidence scoring, protocol layer priority, and an Indeterminate category for ambiguous evidence.
How Voidly correlates three independent measurement projects at scale — data format normalization, 4-hour sliding window alignment, independence-weighted confidence scoring, and handling source disagreements.
Censorship and information control · Machine learning and OSINT
How Voidly probes detect network middleboxes: an HTTP echo test sending custom X-Voidly-Echo headers to a Voidly-controlled server to detect transparent proxies via injected Via/XFF headers, TCP RST injection timing analysis using four heuristics (arrival time, TTL mismatch, zero window, absent TCP options), a vendor signature library with 47 confirmed fingerprints (TSPU/Sandvine/Huawei Hi-SEC/GFW/Cisco), and the middlebox_events TimescaleDB hypertable showing 18-hour median lead time between middlebox detection and censorship anomaly onset across 31 countries.
A deep dive into the TLS layer of Voidly's censorship detection: full certificate chain extraction with rustls, government CA list (China MoI, Iran MICT, Kazakhstan NCA), MITM detection via fingerprint mismatch, TLS alert timing analysis (RST < 15ms = injected), SNI-based blocking detection via dual-SNI probing, ECH/ESNI measurement, and how TLS failure maps to interference_type classifier outputs.
How Voidly built and maintains the 2,300-entry block page fingerprint library used to identify ISP and government censorship block pages: four matching strategies (exact SHA-256 hash, structural normalization, SimHash locality-sensitive hashing, TLS certificate fingerprinting), the match pipeline cascade, block page collection from OONI confirmed events and probe captures, per-country library composition (Turkey 47, Iran 312, Russia 189, China 8), false positive mitigation for CDN error pages and captive portals, and integration with the lf_http_blockpage_hash Snorkel label function.
Censorship and information control · Engineering and infrastructure
How the four Voidly measurement layers compose into a single ProbeResult struct: sequential DNS → TCP → TLS → HTTP execution with the control measurement running in parallel, the None-vs-Some failure propagation convention distinguishing “not attempted” from “attempted and failed”, a failure mode table mapping six layer-outcome combinations to censorship types, and deterministic control vantage selection by domain hash to stabilize body_sha256 comparison across measurement cycles.
A deep dive into the DNS layer of Voidly's censorship detection: dual-resolver design (ISP resolver vs. neutral control), four interference types (NXDOMAIN injection, IP spoofing, empty answer, timeout), the compare_dns_results() algorithm, known injection IP database (China 18 IPs, Iran 3, Turkey 2), CDN geofencing false positive mitigation via ASN group matching, DNSSEC validation limitations, and DoH/DoT diagnostic queries.
How Voidly publishes its measurement corpus to external researchers: a keyset-paginated NDJSON streaming API with (ts, measurement_id) cursor and Server-Sent Events mode, nightly PyArrow Parquet generation sorted by (domain, ts) for 60% I/O reduction on single-domain queries with zstd level-3 compression, atomic HuggingFace Dataset Hub push with dataset card regeneration, and classifier_version tagging to keep probability distributions comparable across model updates.
Censorship and information control · Transparency and open data · Engineering and infrastructure
A complete field-by-field guide to the Voidly CC BY 4.0 measurement dataset — probe identity, DNS/TCP/TLS/HTTP layers, control comparison, ML classification output, BGP signals, corroboration fields, and filtering recipes for journalists and ML researchers.
Censorship and information control · Engineering and infrastructure · Transparency and open data
The three-level TimescaleDB continuous aggregate hierarchy behind Voidly's sub-10ms query latency: measurement_hourly (15-minute refresh), country_daily_summary (1-hour refresh), and country_monthly_stats (daily), cutting a 7-day country query from 4.1 seconds to 4ms. Covers refresh policy configuration, late-arriving probe data handling (94.2% within 1 hour, 98.7% within 24h), compression interplay after 7 days, asn_hourly_summary design, and manual backfill procedures.
Censorship and information control · Engineering and infrastructure
The full path from raw probe bytes to a queryable TimescaleDB record: protobuf over QUIC, Cloudflare Worker validation, Kafka fan-out, Rust normalization, probe-version schema drift handling, quality filtering (3.2% drop rate), and nightly Parquet export to HuggingFace.
Censorship and information control · Engineering and infrastructure
How Voidly ingests BGP data from RIPE NCC RIS, RouteViews, and bgp.tools: MRT format parsing, per-country baseline computation, withdrawal detection thresholds, BgpEvent records in TimescaleDB, and how bgp_outage_score is attached to probe measurements.
Censorship and information control · Engineering and infrastructure
How Voidly uses BGP prefix withdrawal patterns and IODA data to detect internet shutdowns before any probe can send a packet — baseline per-country reachability, the difference between BGP silence and withdrawal, and how BGP fits into the composite confidence score.
Censorship and information control · Engineering and infrastructure
How Voidly uses CAIDA AS-Rank, RIPE NCC RIS route collector data, and PeeringDB to build an AS-level topology, classify censorship choke points (IXP, transit AS, edge ISP), compute per-country probe diversity scores, and feed AS path features into the anomaly classifier.
Censorship and information control · Engineering and infrastructure
How Voidly uses per-ASN probe vantages to distinguish nationwide censorship orders from selective ISP-level blocking — BGP peer classification from CAIDA AS-Rank, ISP blocking fingerprints by interference type, differential blocking detection, and propagation speed analysis that reveals enforcement mechanisms.
How Voidly tracks the full history of blocking events for individual domains across all probe countries — DomainMeasurementSummary continuous aggregate, first/last-seen tracking, the /v1/domains/{domain}/history API, temporal pattern analysis (23% of blocks resolve within 7 days), cross-country blocking correlation, and domain freshness scoring.
Censorship and information control · Engineering and infrastructure
How Voidly aggregates per-measurement interference probabilities into per-country censorship scores: recency decay with a 30-day half-life, ASN diversity weighting, domain category weighting, cross-source corroboration multipliers, 90-day rolling windows, Gaussian temporal smoothing, and bootstrap confidence bands.
Censorship and information control · Engineering and infrastructure
How Voidly aligns OFAC sanctions packages, EU/UN designation timelines, and bilateral diplomatic signals with measured internet shutdown events — building the diplomatic-isolation feature for the shutdown forecasting model.
A deep dive into the feature engineering behind Voidly's 7-day internet shutdown forecasting model: political calendar integration (election dates, protest intensity via GDELT), OFAC sanctions timeline features, BGP withdrawal rate, probe measurement rate drops as precursor signals, historical shutdown patterns, and XGBoost SHAP feature importance across 200 countries.
Censorship and information control · Machine learning and OSINT
How we build a 7-day predictive model for internet shutdowns across 200 countries: political calendar features, network telemetry, ARIMA + XGBoost ensemble, and per-country reliability scoring.
Censorship and information control · Machine learning and OSINT
How Voidly aggregates calibrated per-measurement censorship probabilities into country-level shutdown risk signals: a three-stage aggregation hierarchy (ASN-domain hourly → domain → country), exponential decay weighting with 48-hour half-life over a 14-day window, a 28-feature forecast vector with risk score time series and ASN block concentration, and the Kafka voidly.forecast.features topic handoff to the Bayesian shutdown forecasting service.
Censorship and information control · Machine learning and OSINT
How Voidly calibrates its anomaly classifier separately for each country — Platt scaling logistic regression fitted on per-country holdout predictions, F2-weighted threshold tuning per class, 30-day rolling calibration windows, and calibration case studies: Iran DNS tampering fires at threshold 0.62 (consistent single-authority blocking); China DNS tampering requires 0.74 (CDN split-horizon noise).
Censorship and information control · Machine learning and OSINT
How Voidly retrains its five-class censorship anomaly classifier on a weekly cadence: time-based train/val/test splits to prevent temporal leakage, SMOTE resampling for class imbalance, PSI drift detection, champion/challenger shadow deployment, and the canary rollout process.
Censorship and information control · Machine learning and OSINT
How Voidly serves the anomaly classifier as a live inference API — feature extraction from raw probe measurements in under 5ms, ONNX Runtime for portable model serving, five-class output with per-class probabilities, Cloudflare Worker routing to regional inference nodes, model versioning with champion/challenger shadow mode, and the latency budget that keeps end-to-end probe-to-verdict under 50ms.
Censorship and information control · Machine learning and OSINT · Engineering and infrastructure
How Voidly converts a trained XGBoost censorship classifier to ONNX for serving inside a Rust ingestion service: the sklearn-to-ONNX export pipeline with zipmap=False for zero-copy float32 probability tensors, ONNX Runtime session configuration with per-thread isolation and L3 graph optimization, opset 17 pinning with metadata validation, and batch inference benchmarks achieving p99 under 50ms at batch size 200 on 4 vCPUs.
Censorship and information control · Machine learning and OSINT
How Voidly transforms raw probe measurements into the 47-feature vector that feeds the anomaly classifier: the ControlDelta struct, DNS features (NXDOMAIN injection, bogon IPs, known injection IPs), TCP features (RST timing, SYN-ACK count), TLS features (MITM cert detection, alert codes), HTTP features (blockpage SimHash score, body length ratio), and the LRU control cache design that prevents doubling probe cost.
Censorship and information control · Machine learning and OSINT
How Voidly probes adapt their measurement schedule to device resource constraints: four constraint checks (battery floor, thermal throttle, cellular daily cap, unknown network), sliding-window cellular data accounting with per-minute SQLite buckets, adaptive cycle length that scales domain count to remaining budget via a 28,000-byte-per-measurement estimate, and a priority queue scoring domains on staleness (0.50), config priority flag (0.35), and anomaly recency (0.15).
Censorship and information control · Engineering and infrastructure
How Voidly selects and maintains the domains it probes for censorship: Citizen Lab's global test list, 12 OONI category codes, per-country supplemental lists, the measurement budget problem, and why the test list is a political document.
Censorship and information control · Engineering and infrastructure
How Voidly protects probe operators in jurisdictions that criminalize censorship measurement: strict data minimization (no name, address, or IP logging), WireGuard peer-key authentication, daily probe ID pseudonymization, optional Tor hidden service upload, measurement scrubbing, country-tier legal risk assessments, and a one-tap emergency stop with full data erasure.
Censorship and information control · Cybersecurity and privacy · Engineering and infrastructure
How a new Voidly probe operator goes from application to publishing measurements: on-device X25519 key generation in the Tauri app, probe registration and ASN verification, 48-hour warmup period with calibration measurements, quality scoring at promotion, and what happens when warmup calibration fails.
Censorship and information control · Engineering and infrastructure
How Voidly selects and distributes its probe vantage network: why ASN diversity matters more than geographic spread, the operator safety constraints that shape high-risk country probes, and how we reach countries where most people connect on mobile-only networks.
Censorship and information control · Engineering and infrastructure
How Voidly delivers measurement configuration to probes without a persistent control channel: gzip+CBOR bundles signed with Ed25519 (signature verified before decompression to prevent zip-bomb attacks), a pull-based auto-update scheduler with 6-hour intervals and exponential backoff, version pinning and two-snapshot rollback, and anonymous country tokens derived via BLAKE3 from ISO code + epoch-week salt so the CDN cannot correlate which overlay a probe applies.
Censorship and information control · Engineering and infrastructure · Cybersecurity and privacy
How Voidly protects probe operator identity while publishing full measurement data: probe_id derived as SHA-256(public_key_bytes) with zero IP logging, human-readable codename system (450K+ combinations, no joint table with probe_id), measurement anonymization (probe_cc + probe_asn published; IP never stored), per-probe Ed25519 signing with isolated key store, and 12-country extra protections (4–48 hour publication delay, 90-day probe_id rotation).
Censorship and information control · Engineering and infrastructure
How Voidly manages storage for 2.2B probe measurements using a three-tier TimescaleDB retention policy — full-resolution hot tier (0-30 days), native-compressed warm tier (31-365 days, 6.2x ratio), and downsampled cold tier (>365 days, aggregates only), with continuous aggregate cascade, pg_cron compliance verification, and R2 tiered storage planned for Q3 2026.
Censorship and information control · Engineering and infrastructure
How Voidly stores and queries 2.2 billion censorship probe results in TimescaleDB: hypertable design with 1-day chunk intervals and secondary country partitioning, 6.2× compression, continuous aggregates for country-level daily summaries, three-tier retention (hot/warm/cold), and query benchmarks for anomaly detection.
Censorship and information control · Engineering and infrastructure
How Voidly's corroboration engine fetches and aligns data from three independent sources in near-real-time despite their different latency profiles: tokio::join! parallel fetches with per-source timeouts, adaptive OONI polling (15m/60m/3h/6h), in-memory CensoredPlanet daily dump index, independence-weighted source agreement scoring, and retroactive nightly reprocessing against the CP daily dump.
Censorship and information control · Engineering and infrastructure
How the Voidly MCP server exposes 83 tools for querying the global censorship dataset from Claude, GPT, and agent frameworks — incident lookup, measurement queries, country summaries, BGP events, shutdown forecasts, and wiring it into Claude Code.
Censorship and information control · Engineering and infrastructure
How the nightly Voidly export job extracts measurements from TimescaleDB and pushes Parquet snapshots to HuggingFace Hub: PyArrow schema with dictionary-encoded columns, server-side cursor streaming at 50K rows per round-trip, Zstandard level 3 compression, country + year_month partitioning, atomic HuggingFace commit with CommitOperationAdd, post-push SHA-256 verification, and the incremental vs. monthly full-snapshot strategy.
Censorship and information control · Engineering and infrastructure · Transparency and open data
How the Voidly CC BY 4.0 measurement dataset and the OONI historical corpus are hosted on HuggingFace — Parquet snapshot structure, daily incremental updates, git-lfs versioning, and Python/R filter recipes for journalists, ML researchers, and infrastructure teams.
Censorship and information control · Transparency and open data · Engineering and infrastructure
How a Voidly censorship incident progresses through six states — Anomaly, MultiSourceAnomaly, Corroborated, VerifiedIncident, Resolved, FalsePositive — with exact transition thresholds, timing data from 847 incidents in 2024 (67% stuck at Anomaly, 18% reach VerifiedIncident), IncidentRecord struct, publication timing by tier, how lifecycle state encodes into HuggingFace dataset fields, and retroactive state change handling via incident_history.
How a Voidly measurement moves through three confidence tiers — Anomaly, Corroborated, Verified Incident — and what each tier means for journalists, ML researchers, and infrastructure monitoring teams using the dataset.
Censorship and information control · Engineering and infrastructure
How the Voidly SSE streaming endpoint delivers censorship events in real time: GET /v1/stream with country/tier/type filtering, four event types (incident_created, incident_updated, incident_resolved, country_status_change), Last-Event-ID reconnection with 24-hour event ring buffer, Python httpx.Client and JavaScript EventSource examples, and how SSE differs from the webhook delivery system.
Censorship and information control · Engineering and infrastructure
How the Voidly REST API is designed: key endpoints for incident lookup, measurement queries, country summaries, domain history, BGP events, and 7-day shutdown forecasts; cursor-based pagination, filtering, rate limits, and code samples in curl, Python, and JavaScript.
Censorship and information control · Engineering and infrastructure
How the Voidly API handles authentication: two access tiers (public 60 req/hr and keyed), voidly_{env}_{base58} key format with PBKDF2-HMAC-SHA256 storage, D1 + KV request authentication flow, four plan tiers (Free/Research/Professional/Enterprise), HMAC-SHA256 webhook signature verification, key rotation without downtime, test keys for CI, and OAuth2 for third-party integrations.
Censorship and information control · Engineering and infrastructure
How Voidly gets verified censorship incidents to journalists, researchers, and monitoring systems: HMAC-signed webhook delivery with exponential-backoff retry, PGP-encrypted email for verified alerts, per-country and per-confidence-tier RSS feeds, alert deduplication by incident_id, and rate-limiting to prevent fatigue.
Censorship and information control · Engineering and infrastructure
How Voidly transitions a censorship incident through five states (Anomaly/MultiSourceAnomaly/Corroborated/Verified/Resolved) with threshold-gated transitions, stores every state change as an append-only event in a TimescaleDB hypertable with SHA-256 idempotency_key, and fans out verified incidents to alert delivery and cache invalidation via three Kafka topics — with the compute_incident_id() Rust function that makes incident IDs deterministic across pipeline restarts.
Censorship and information control · Engineering and infrastructure
How Voidly schedules 80-domain probe runs across 37+ nodes: domain priority tiers by OONI category code, anomaly-driven priority boosts, protocol selection per domain, ±15% jitter for anti-detection, ASN distribution to ensure cross-ASN coverage, adaptive scheduling that injects urgent re-measurements on anomaly detection, and per-country task budgets (CN 68, IR 74, RU 72, global avg 49 tasks/window).
Censorship and information control · Engineering and infrastructure
How Voidly ingests 200M+ OONI Explorer measurements, aligns them with Voidly probe data on a country-domain-date key, generates probabilistic training labels using five Snorkel-style label functions, handles OONI coverage gaps with label distillation, and constructs the labeled dataset that trains the five-class anomaly classifier.
Censorship and information control · Machine learning and OSINT
How the Voidly ML classifier distinguishes DNS tampering, TLS interference, HTTP blocking, BGP withdrawal, and throttling — five per-class binary models, country-specific calibration, and why 95% recall beats 95% precision when cross-source corroboration filters the noise.
Censorship and information control · Machine learning and OSINT · Engineering and infrastructure
How Voidly evaluates the five-class censorship anomaly classifier offline before deployment: the ClassifierEvaluator test harness, per-country AUC-PR vs. AUC-ROC tradeoffs for imbalanced censorship data, F2 scoring rationale, per-country confusion matrix case studies (Iran 0.97 DNS recall, China 0.78 precision from CDN noise, Russia TSPU throttling), ECE calibration before and after Platt scaling, and model promotion criteria.
Censorship and information control · Machine learning and OSINT
How Voidly uses uncertainty sampling, Cohen's kappa inter-annotator agreement, and weekly model retrains to grow its censorship anomaly training set from 127K bootstrap labels to 275K — 500 examples/week annotated by 3 researchers each, with DVC data versioning and PSI drift detection.
Machine learning and OSINT · Censorship and information control
How Voidly constructs a labeled training dataset for the anomaly classifier from 200M+ OONI measurements: weak supervision with Snorkel-style label functions across DNS/TCP/TLS/HTTP layers, class imbalance handling with SMOTE and log-weighting, time-based train/val/test splits to prevent leakage, per-country Platt scaling calibration, and the continuous retraining pipeline.
Censorship and information control · Machine learning and OSINT · Engineering and infrastructure
How the quality filter pipeline decides which raw measurements are fit for ML training: boolean checks for control_failure (1.9% drop rate — ISP blocks on control server IPs in CN/IR/RU), missing_fields (0.8%), old probe version pre-2.5.0 (0.3%), and duplicates (0.2%), totalling 3.2% dropped. Includes the quality_filter() Python function, the to_feature_input() schema transformation, and why rejected measurements go to quarantine not discard.
Censorship and information control · Machine learning and OSINT · Engineering and infrastructure
How Voidly normalizes 200M+ OONI measurements across five web_connectivity schema versions (v0.2 to v0.6) into a single ML-ready format: a detect_web_connectivity_version() function using field-presence inference, AnomalyType and ConfidenceTier enums, the OoniMeasurementNormalized dataclass, FLAG_* bitmask constants for DNS/TCP/TLS/HTTP anomaly encoding, side-by-side normalize_v05() vs. normalize_v06() implementations, and a 95.3% pass-through rate from the drop-reason table.
Censorship and information control · Engineering and infrastructure · Machine learning and OSINT
How we processed the OONI raw measurement archive into a flat ML-ready CSV: handling probe version schema drift across 12 years, normalizing test_keys across 20 measurement types, streaming 200M+ records, and what we decided to leave out.
Censorship and information control · Engineering and infrastructure
How Voidly attributes censorship infrastructure to specific DPI vendors using network signatures and open-source intelligence: a six-vendor signature table (TSPU/Sandvine/NetClean/Iran ARRS/Cisco IronPort/GFW), DpiVendorSignature dataclass with a score_signature_match() function weighting RST timing (0.35), block page (0.30), injection IP (0.25), and CA SPKI (0.10), procurement scraping with PROCUREMENT_SOURCES across five government tender portals, BGP TTL-hop attribution, and case studies for Russia, Iran, and Ethiopia.
Censorship and information control · Machine learning and OSINT · Engineering and infrastructure
How Voidly identifies the hardware and software responsible for internet censorship: blocking architecture taxonomy (L3/L4/L7-DNS/L7-HTTP), DPI vendor signatures from timing patterns (Russia's TSPU RST < 3ms, Iran's ARRS DNS injection IPs, China's GFW TTL fingerprinting), ISP-level blocking fingerprints (Rostelecom vs. MTS vs. Turkcell), TTL analysis for middlebox distance, OSINT cross-referencing with procurement records, and the censorship_infrastructure dataset field.
Censorship and information control · Machine learning and OSINT
How we built a censorship-resistant VPN for Voidly probe operators: GFW/IRGC/TSPU threat model, WireGuard inside HTTP/2 CONNECT domain-fronting over CDN edges, 48hr entry-node IP rotation via Cloudflare KV, traffic morphing (Laplace timing jitter + packet-size CDF matching + cover traffic), 22-dim XGBoost on-device routing with ONNX, BlockageDetector for RST injection, and 99.3% DPI evasion across CN/IR/RU.
The statistical methods behind AI Analytics' election anomaly detection — first-digit analysis, last-digit uniformity testing, turnout z-scores, and why these signals require cross-validation with social and media data before generating an alert.
Money in politics · Machine learning and OSINT · Censorship and information control
All topics: the writing index.