Writing · topic · 117 articles
How the datasets and systems are built: pipelines, storage, APIs, and measurement and serving infrastructure.
The 2025 federal inventory lists 3,611 agency-reported AI use cases and 445 designated high-impact systems. For 318 high-impact records, the agency left pre-deployment testing unstated; only 16 report an established appeal process. The blanks are disclosure facts, not proof a safeguard is absent — and the inventory spans every lifecycle stage, not only deployed systems.
Engineering and infrastructure · Transparency and open data · Federal data
The United States runs on 1.38 terawatts of generating capacity, and the federal filings say who owns every megawatt. Computed from Form EIA-860: independent power producers now out-own the investor-owned utilities on your bill; the federal government is one of the largest owners in the country; and a quarter-terawatt is jointly owned through capacity shares most customers have never heard of.
Ownership and consolidation · Environment and energy · Engineering and infrastructure · Transparency and open data
There is no single US cyber-breach registry. One incident can surface in three unconnected federal places — CISA’s KEV catalog (the exploited vulnerability), an SEC 8-K Item 1.05 filing (the material event), and the HHS OCR breach portal (health data) — each with a different trigger, threshold, and clock. A guide to joining them by victim organization and date.
Cybersecurity and privacy · Finance and markets · Transparency and open data · Engineering and infrastructure
There is no single US recall database — cars are recalled by NHTSA, consumer products by the CPSC, food/drugs/devices by the FDA, and meat and poultry by USDA-FSIS. A guide to weaving the four feeds into one cross-agency recall view, joined on firm name and date, with the hazard-classification mismatches and what a unified view reveals.
Consumer protection · Health and medicine · Transportation safety · Engineering and infrastructure
Three public datasets describe how organizations try to shape government and what they receive: FEC campaign finance (who gives), lobbying disclosures (who lobbies), and USAspending (who wins contracts). Joined on organization + parent name — there is no shared identifier — with the honest correlation-not-causation caveat.
Money in politics · Government operations · Transparency and open data · Engineering and infrastructure
A tax-exempt organization’s full federal footprint — its IRS exemption ruling, its self-reported Form 990 finances and grants made, and the USAspending grants, contracts, and subawards flowing to it — joined on the EIN, the universal nonprofit key. The money-in vs money-out distinction and the gotchas that break the join.
Transparency and open data · Government operations · Engineering and infrastructure
Federal money funds research, that research becomes patents, and some patents become products. Four datasets follow the path — NIH and NSF grants, USPTO patents (whose Bayh-Dole government-interest statements disclose the funding behind them), and FDA approvals — joined on institution, inventor, and the government-interest clause.
Six federal datasets follow a prescription drug across its life — FDA approval, the National Drug Code directory, CMS Open Payments (manufacturer payments to prescribers), Medicare Part D prescribing and spending, and CDC overdose mortality — joined on the NDC code, ingredient, and manufacturer. The keys, the brand/generic and NDC-format gotchas, and what the assembled pipeline answers.
Health and medicine · Transparency and open data · Engineering and infrastructure
A structured, source-cited index of book censorship worldwide — 19,283 banned or restricted titles across 119 countries — built on the CC-BY banned-books.org Open Censorship Core and served as static JSON for AI agents to query: is this book banned in that country, and why?
Censorship and information control · Transparency and open data · Engineering and infrastructure
US trade controls and the trade they govern usually live in separate datasets — OFAC’s sanctions lists, BIS’s export-enforcement record, and the Census foreign-trade statistics. This guide joins all three by country, HS commodity, and party so the rules, the violations, and the actual flow of goods line up in one view.
Economy and demographics · Sanctions and illicit finance · Engineering and infrastructure
A guide to building a single facility view from four EPA compliance datasets — joining the RCRA hazardous-waste registry, the Toxic Release Inventory, Clean Air Act compliance, and the ECHO enforcement record through the Facility Registry Service ID, so a site’s permits, pollution, violations, and penalties come together in one place. Covers the join key, the program identifiers, the environmental-justice questions the assembled data answers, and a worked Python walkthrough against EPA’s key-free public APIs.
Four federal hazard datasets — the National Weather Service alert feed, NOAA’s Storm Events Database, the USGS earthquake catalog, and FEMA’s disaster declarations — line the warning, the event, the damage, and the federal response up in one view. A guide to joining them on place and time so an analyst can measure warning lead time against casualties and see what share of damaging events ever became a federal disaster.
The US food-safety system is split across three federal agencies — CDC epidemiology, USDA FSIS recalls of meat and poultry, and FDA recalls of everything else — and the chain from a detected outbreak to the recall that pulls contaminated food off shelves only comes together when their datasets are joined. This guide traces that chain through public, key-free data, aligning the records by pathogen, product, firm, and time.
Food and agriculture · Health and medicine · Engineering and infrastructure
Four federal energy datasets — EIA’s plant and generator inventory, the ownership schedule, the electricity generation and price series, and FERC’s market-manipulation enforcement record — joined on the plant code and the operator name into one view of the physical grid, who owns it, what it produces, and who polices it.
A guide to mapping the federal research enterprise from data — joining NSF awards, NIH grants, and ORI misconduct findings on institution and investigator so the flow of federal science money and its oversight come together in one view. Covers the two funding pillars, the misconduct accountability layer, the join-key normalization problem, and a Python workflow that aggregates funding and lines findings up against grants.
Whether a given college is worth it can be answered school by school — and now program by program — by joining two federal datasets: NCES IPEDS, the mandatory census of what every college charges and graduates, with the College Scorecard’s record of debt and post-enrollment earnings. A guide to the UnitID/OPEID join, the CIP program layer, and a worked earnings-to-debt value ratio.
A guide to tracing the employment-based immigration pipeline through federal data — joining the Department of Labor’s foreign-labor certifications, the USCIS record of H-1B petitions, and the BLS wage statistics that set the prevailing-wage floor, so the path by which a US employer hires a foreign worker comes together in one view.
Justice and immigration · Labor and workplace · Engineering and infrastructure
A guide to tracing a federal regulation end to end — from the public law that authorizes it, through the proposed and final rules in the Federal Register, to the public-comment docket on Regulations.gov — by threading the RIN, the docket ID, and the Federal Register document number across three connected datasets.
A single workplace’s federal safety history is scattered across four OSHA datasets — the inspections, the citations they generate, the severe-injury reports employers must file, and the annual Form 300A summaries. This guide assembles them into one establishment-level view, keyed by inspection number and employer, and asks whether enforcement actually lowers injury rates afterward.
Labor and workplace · Federal data · Engineering and infrastructure
A guide to reconstructing the full arc of a US catastrophe from four OpenFEMA datasets — joining the disaster declaration, the Public Assistance grants that rebuild, the Hazard Mitigation projects that prevent the next loss, and the NFIP flood claims that pay homeowners — so one hurricane, wildfire, or flood can be followed from the day the President signs the declaration through every dollar the federal government spends in response.
Most federal data syntheses fail at the join — there is no shared key, so the analyst is left fuzzy-matching names. The SEC ecosystem is the rare exception: EDGAR’s Central Index Key uniquely identifies every filer, so the company registry, the 8-K material-event filings, the 13F holdings that name the company, its Form D private placements, and its litigation and administrative enforcement all link without ambiguity. This guide assembles all six SEC datasets into one CIK-keyed corporate profile that follows a company across its entire public life.
No single federal dataset shows the opioid epidemic whole — but three of them, joined on geography, do. This guide aligns the DEA’s ARCOS record of how the pills were shipped, the CDC’s death-certificate record of how people died, and the CMS record of where Medicare addiction treatment reaches survivors, so the supply, the toll, and the response line up in one geographic view.
A bank rarely fails without warning — its slide shows up in the quarterly call reports quarters before regulators close it. This is a guide to reconstructing the whole arc from federal data, joining the FDIC institution registry, the call-report financials, the enforcement orders, and the failed-bank record on one clean key: the FDIC certificate number.
Finance and markets · Federal data · Engineering and infrastructure
A single agency owns the entire defect-to-outcome loop — NHTSA collects the complaints, opens the investigations, compels the recalls, tracks the completion reports, and counts the deaths. This guide assembles all five datasets into one pipeline and shows how to join a complaint cluster to the recall it provoked and the fatalities it cost.
Transportation safety · Consumer protection · Engineering and infrastructure
The US aviation-safety feedback loop is split across two agencies — the NTSB that investigates and the FAA that regulates — so the accident finding and the mandatory fix live in different databases. This guide traces an unsafe condition from accident, to airworthiness directive, to the registered fleet it applies to, joining four federal datasets on aircraft make and model into one accountable chain.
No single federal file tells you how Americans die — five CDC mortality datasets do, when you assemble them: the leading-causes ranking, the injury and external-cause records, the suicide series, drug-overdose mortality, and excess deaths. All are cut from the same National Vital Statistics System death certificates, so they share join keys and age-adjustment, and the work is aligning their cause definitions rather than parsing five separate NCHS releases.
Consolidation is the defining force in American healthcare, but it leaves its fingerprints across four separate CMS records that do not share a key. This is a field guide to bridging the provider-ownership files, the change-of-ownership transactions, and the Care Compare quality datasets through the enrollment-to-CCN crosswalk — turning who owns a facility, the deal that changed it, and the staffing and outcomes that followed into one traceable, facility-level story.
Health and medicine · Ownership and consolidation · Engineering and infrastructure
The hardest and most valuable move in federal data analysis is following one company across four disconnected systems — the political money it gives, the lobbying it pays for, the contracts it wins, and the fraud cases brought against it. None of them share a key, so the work is entity resolution: normalizing names, mapping subsidiaries to parents, and living with fuzzy matches.
Engineering and infrastructure · Money in politics · Government operations
The EPA’s inventory of every public water system in the country — an inventory file of roughly 400,000 records keyed by PWSID, of which about 150,000 are active systems, from the largest city utilities to the campground wells that serve twenty-five people. A field-level guide to system types, size categories, source water, state primacy, and the inventory that the violations and inspection datasets all hang off of.
Environment and energy · Engineering and infrastructure · Federal data
The Federal Highway Administration Highway Performance Monitoring System is the national database for US roadway conditions — collecting pavement condition ratings, traffic volumes, lane miles, and functional class data for 4.1 million miles of public roads, from Interstate highways to rural local roads, enabling Congress to calculate federal highway funding formulas and researchers to track infrastructure decline.
Transportation safety · Engineering and infrastructure · Federal data
The Pipeline and Hazardous Materials Safety Administration maintains incident reports for every significant gas and liquid pipeline accident in the United States — spills, explosions, injuries, fatalities, and property damage — creating the most comprehensive public record of pipeline safety performance across 2.7 million miles of US pipeline infrastructure.
Transportation safety · Engineering and infrastructure · Federal data
The Federal Highway Administration National Bridge Inventory collects biennial condition ratings for every highway bridge in the United States — 620,000 bridges covering structural sufficiency, deck ratings, superstructure, substructure, and channel conditions.
Transportation safety · Engineering and infrastructure · Federal data
The Federal Highway Administration publishes the most comprehensive infrastructure dataset in the federal government: the National Bridge Inventory (620,000+ bridges, biennial inspection, 0–9 condition ratings, sufficiency score), the Highway Performance Monitoring System (pavement IRI, Good/Fair/Poor condition, 900,000+ road segments), Annual Average Daily Traffic counts, and Highway Statistics (registered vehicles, licensed drivers, gas tax revenues). Here is the structurally deficient vs. functionally obsolete distinction, the IIJA 2021 $40B bridge repair program, the Highway Trust Fund solvency crisis (gas tax frozen at $0.184/gallon since 1993, EVs avoiding it), the Freight Analysis Framework commodity-flow OD matrices, and a Python NBI bridge data script to map structurally deficient bridges by sufficiency rating.
Transportation safety · Engineering and infrastructure · Federal data
How we built a 0–100 compliance risk score across OFAC, SAM, OIG, CFPB, SEC, DOJ, FDIC, FINRA, CFTC, EPA, MSHA, FDA warning letters, PCAOB, UFLPA, and 15+ more lists in a single API call.
Federal data · Sanctions and illicit finance · Engineering and infrastructure
How the Federal Regulatory Data Hub resolves entity identity across 30+ compliance lists: three-stage pipeline (identifier join 34%, FTS5 canonical name 41%, Jaro-Winkler fuzzy 18%), false positive taxonomy (same-name different entity 47%, subsidiary-parent 28%, historical name 16%, transliteration 9%), EntityResolutionResult confidence-to-action mapping (MATCH ≥0.90, PROBABLE_MATCH 0.72–0.90), 99.1% recall, 98.7% precision at ≥0.90, and weekly analyst-feedback calibration loop.
Federal data · Machine learning and OSINT · Engineering and infrastructure
How the Federal Regulatory Data Hub resolves entity names across 208 federal datasets when identifiers disagree — OFAC alias explosion (44K aliases from 12K entries), SEC EDGAR subsidiary mapping, three-pass fuzzy matching (exact → Jaro-Winkler → TF-IDF cosine), 1.4% combined false positive rate, and how entity_confidence weights the compliance risk score.
How the Federal Regulatory Data Hub generates and maintains stable canonical IDs for entities across 208 federal datasets — deterministic SHA-256 ID generation, EntityVersion history for merge and split events, EntityAlias tracking for historical name variants, and subscriber continuity guarantees when source identifiers change.
How we built an entity bridge across 208 federal datasets so a single query returns every SEC filing, FDA warning letter, EPA enforcement case, and OFAC sanction for any company.
How the Federal Regulatory Data Hub lets compliance teams subscribe to regulatory events for specific entities — using the cross-agency entity bridge to watch OFAC, SAM, SEC, EPA, DOJ, and 25+ other lists simultaneously.
How the Federal Regulatory Data Hub detects regulatory record changes and delivers them to subscribers: 10-minute OFAC sanctions window, 30-minute SAM debarment window, EDGAR 8-K filing webhooks, HMAC-signed Cloudflare Queue delivery with at-least-once semantics, per-entity and per-list subscription filters, and idempotency_key deduplication.
How we ported the Swarm SDK cryptographic core to no_std Rust targeting the STM32H7 Cortex-M7: feature-gated std/embedded builds, 96KB static heap with cortex-m-alloc, pre-allocated VecDeque deduplication ring, in-place AES-GCM to avoid heap allocation, hardware AES accelerator integration (0.14ms vs. 0.61ms software), and binary size optimization from 1.2MB to 284KB with opt-level="z" and LTO.
How the Federal Regulatory Data Hub exposes its data through an MCP server with 38+ tools for Claude, GPT, and other AI agents — screen_entity, get_entity, compliance reporting tools, HMAC-signed webhook configuration, rate-limit tiers by plan, and Claude Desktop integration via stdio transport.
How the Federal Regulatory Data Hub API is designed: no-auth CC0 REST endpoints, cross-agency entity resolution in a single GET, an MCP server with 38+ tools for Claude and GPT agent workflows, and JSON-LD structured data for search indexing.
How we ingest and refresh 208 federal regulatory datasets across 45 agencies using Cloudflare Workers cron, delta detection, schema drift handling, and per-source retry budgets — the ETL behind the Federal Regulatory Data Hub.
How the Swarm SDK MeshTransport layer achieves reliable frame delivery over lossy drone radio links: sliding window ARQ with selective ACK, EWMA RTT estimation, transparent fragmentation and reassembly for Sealed Sender envelopes, multi-channel bonding across 2.4GHz and 5.8GHz radios, and performance benchmarks on STM32H7 and Jetson Nano.
How the Swarm SDK implements a gossip mesh for drone swarms: epidemic broadcast with k=3 fanout, UUIDv4 sliding-window deduplication across a 1000-ID VecDeque, Lamport clock causal ordering for key management messages, TTL hop limiting with 3-hop lossy-channel headroom, and anti-entropy reconciliation for post-partition recovery — with STM32H7 and Jetson Nano benchmarks.
How Voidly deduplicates thousands of probe measurements into discrete censorship incidents: the four-tuple clustering key, the 6-hour gap rule, incident lifecycle from ANOMALY to RESOLVED, incident_id assignment, retroactive CensoredPlanet alignment, and edge cases including flapping blocks and BGP outages.
Censorship and information control · Engineering and infrastructure
How Voidly reconstructs the authoritative timeline of a censorship incident from asynchronous distributed probe measurements — IncidentEvent sourcing model, temporal alignment across time zones, confidence weighting requiring 3+ independent probes, retroactive revision from CensoredPlanet batch data, duration statistics, and the timeline REST API endpoint.
Censorship and information control · Engineering and infrastructure
How Voidly determines that a censorship incident has ended: per-type resolution thresholds (consecutive passing measurements with p_blocked < 0.3), the 12-hour RESOLVED_PENDING re-open window, FLAPPING state detection for rapidly alternating blocks, BGP-type auto-resolution, and cross-source confirmation requirements for VERIFIED incidents — with observed resolution time distributions (BGP 4.2h median, HTTP 12.1 days).
Censorship and information control · Engineering and infrastructure
How Voidly embeds ONNX Runtime inside an Apache Flink streaming job to score probe results for censorship anomalies at 50,000 events/sec with sub-100ms end-to-end latency: thread-local ONNX session management per task slot, Kafka partition alignment with (country_code, asn) keyBy, mini-batch coalescing for 50ms p99 inference, and the backpressure mechanism that keeps consumer lag under 2,400 messages even on election-day traffic spikes.
Censorship and information control · Engineering and infrastructure · Machine learning and OSINT
How Voidly gets from a probe anomaly to a published verified incident — and an alert in a journalist's inbox — in under 8 minutes: the event queue, real-time OONI and IODA API polling, confidence threshold crossing, the two-window alert-fatigue guard, and the nightly CensoredPlanet retroactive pass.
Censorship and information control · Engineering and infrastructure
What happens inside a single Voidly probe run: the measurement execution loop, DNS and TCP and TLS and HTTP data capture, result serialization and signing, and the upload path that delivers a signed ProbeResult to the ingest pipeline.
Censorship and information control · Engineering and infrastructure
How Voidly probes maintain connectivity and upload measurements from networks that actively block VPN protocols — QUIC/443 transport, domain fronting via CDN SNI fronting, TLS certificate pinning against MITM, local SQLite buffering (500 MB cap, 48h window), and metered-connection backoff.
Censorship and information control · Engineering and infrastructure
How Voidly probes preserve measurement data during upload failures — a 72-hour SQLite ring buffer with anomaly-safe eviction, LZ4 batch compression reducing median batch size from 47KB to 9KB, exponential backoff retry up to 4 hours, priority queue for anomalous measurements, chunked upload with per-chunk acknowledgment, and 0.003% measurement loss rate across 37 probes over 6 months.
Censorship and information control · Engineering and infrastructure
How the Voidly desktop probe works: Tauri 2 cross-platform app, Cloudflare boringtun WireGuard, tun-rs TUN device, X25519-Dalek on-device key generation, and operator anonymity as a design constraint.
Censorship and information control · Engineering and infrastructure
How the Voidly probe test runner orchestrates concurrent measurements inside the Tauri app: tokio Semaphore with 3 permits, MeasurementState machine (Pending → Running → Success/Error/Timeout), per-layer timeout budgets (DNS 3s, TCP 5s, TLS 8s, HTTP 15s, total 30s), Ed25519 measurement signing, mpsc upload queue with capacity 200, and why per-layer timeouts are themselves evidence of DNS-layer interference.
Censorship and information control · Engineering and infrastructure
A step-by-step breakdown of how each Voidly probe test works: DNS resolution, TCP handshake, TLS negotiation with certificate chain validation, HTTP request execution, response body fingerprinting, control comparison, and how every layer maps to interference types in the anomaly classifier.
Censorship and information control · Engineering and infrastructure
How Voidly uses a distributed control server network to distinguish genuine censorship from network errors, CDN split-horizon DNS, and misconfigured sites — DNS, TCP, TLS, and HTTP comparison methodology, and why a single control is not enough.
Censorship and information control · Engineering and infrastructure
A technical deep-dive on how Voidly detects bandwidth throttling — the hardest interference class to classify. Covers the TimingFeatures Rust struct, TTFB z-score computation against control measurements, body truncation and mid-transfer RST signals, the congestion vs. deliberate-throttling calibration problem, cross-probe corroboration scoring, and country patterns from Russia TSPU, Iran ARRS, India, and China.
Censorship and information control · Engineering and infrastructure
How Voidly monitors 37+ probe nodes: heartbeat system (60s cadence, separate transport), DEGRADED/OFFLINE state machine, measurement quality scoring, ASN coverage SLOs for 200 countries, flapping detection capping confidence at CORROBORATED, automated replacement from standby operator waitlist, and the classify_offline_cause() algorithm distinguishing probe failure from ISP-level censorship.
Censorship and information control · Engineering and infrastructure
How Voidly probes identify DNS injection and manipulation in censored networks — comparison against three control resolvers, four weighted detection signals (IP divergence, TTL anomaly, source IP divergence, response timing), per-country injection rates (China 94%, Iran 61%, Russia 12%), CAP_NET_RAW privilege handling, anycast false-positive calibration from 4.2% to 0.8%, and integration with the DnsTestResult confidence score.
Censorship and information control · Engineering and infrastructure
How Voidly avoids false positives from commercial geoblocking: HTTP 451 detection, streaming service block page fingerprints (tagged geoblock_commercial, not censorship), multi-country probe comparison (SINGLE_COUNTRY vs. MULTI_COUNTRY_SELECTIVE geographic patterns), CDN split-horizon detection via ASN group mapping, domain-level unavailability baselines, and the p_geoblock score that suppresses measurements above 0.70.
Censorship and information control · Engineering and infrastructure
How Voidly built and maintains the 2,300-entry block page fingerprint library used to identify ISP and government censorship block pages: four matching strategies (exact SHA-256 hash, structural normalization, SimHash locality-sensitive hashing, TLS certificate fingerprinting), the match pipeline cascade, block page collection from OONI confirmed events and probe captures, per-country library composition (Turkey 47, Iran 312, Russia 189, China 8), false positive mitigation for CDN error pages and captive portals, and integration with the lf_http_blockpage_hash Snorkel label function.
Censorship and information control · Engineering and infrastructure
How Voidly publishes its measurement corpus to external researchers: a keyset-paginated NDJSON streaming API with (ts, measurement_id) cursor and Server-Sent Events mode, nightly PyArrow Parquet generation sorted by (domain, ts) for 60% I/O reduction on single-domain queries with zstd level-3 compression, atomic HuggingFace Dataset Hub push with dataset card regeneration, and classifier_version tagging to keep probability distributions comparable across model updates.
Censorship and information control · Transparency and open data · Engineering and infrastructure
A complete field-by-field guide to the Voidly CC BY 4.0 measurement dataset — probe identity, DNS/TCP/TLS/HTTP layers, control comparison, ML classification output, BGP signals, corroboration fields, and filtering recipes for journalists and ML researchers.
Censorship and information control · Engineering and infrastructure · Transparency and open data
The three-level TimescaleDB continuous aggregate hierarchy behind Voidly's sub-10ms query latency: measurement_hourly (15-minute refresh), country_daily_summary (1-hour refresh), and country_monthly_stats (daily), cutting a 7-day country query from 4.1 seconds to 4ms. Covers refresh policy configuration, late-arriving probe data handling (94.2% within 1 hour, 98.7% within 24h), compression interplay after 7 days, asn_hourly_summary design, and manual backfill procedures.
Censorship and information control · Engineering and infrastructure
The full path from raw probe bytes to a queryable TimescaleDB record: protobuf over QUIC, Cloudflare Worker validation, Kafka fan-out, Rust normalization, probe-version schema drift handling, quality filtering (3.2% drop rate), and nightly Parquet export to HuggingFace.
Censorship and information control · Engineering and infrastructure
How Voidly ingests BGP data from RIPE NCC RIS, RouteViews, and bgp.tools: MRT format parsing, per-country baseline computation, withdrawal detection thresholds, BgpEvent records in TimescaleDB, and how bgp_outage_score is attached to probe measurements.
Censorship and information control · Engineering and infrastructure
How Voidly uses BGP prefix withdrawal patterns and IODA data to detect internet shutdowns before any probe can send a packet — baseline per-country reachability, the difference between BGP silence and withdrawal, and how BGP fits into the composite confidence score.
Censorship and information control · Engineering and infrastructure
How Voidly uses CAIDA AS-Rank, RIPE NCC RIS route collector data, and PeeringDB to build an AS-level topology, classify censorship choke points (IXP, transit AS, edge ISP), compute per-country probe diversity scores, and feed AS path features into the anomaly classifier.
Censorship and information control · Engineering and infrastructure
How Voidly tracks the full history of blocking events for individual domains across all probe countries — DomainMeasurementSummary continuous aggregate, first/last-seen tracking, the /v1/domains/{domain}/history API, temporal pattern analysis (23% of blocks resolve within 7 days), cross-country blocking correlation, and domain freshness scoring.
Censorship and information control · Engineering and infrastructure
How Voidly aggregates per-measurement interference probabilities into per-country censorship scores: recency decay with a 30-day half-life, ASN diversity weighting, domain category weighting, cross-source corroboration multipliers, 90-day rolling windows, Gaussian temporal smoothing, and bootstrap confidence bands.
Censorship and information control · Engineering and infrastructure
How the Federal Regulatory Data Hub ingests the OFAC Specially Designated Nationals list — daily conditional GET with ETag, XML parsing across 12K SDN entries with alias explosion, name normalization pipeline, FTS5 + Jaro-Winkler three-pass screening, and p50 8ms / p99 28ms screening latency against the SDN list alone.
Federal data · Sanctions and illicit finance · Engineering and infrastructure
How Voidly serves the anomaly classifier as a live inference API — feature extraction from raw probe measurements in under 5ms, ONNX Runtime for portable model serving, five-class output with per-class probabilities, Cloudflare Worker routing to regional inference nodes, model versioning with champion/challenger shadow mode, and the latency budget that keeps end-to-end probe-to-verdict under 50ms.
Censorship and information control · Machine learning and OSINT · Engineering and infrastructure
How Voidly probes adapt their measurement schedule to device resource constraints: four constraint checks (battery floor, thermal throttle, cellular daily cap, unknown network), sliding-window cellular data accounting with per-minute SQLite buckets, adaptive cycle length that scales domain count to remaining budget via a 28,000-byte-per-measurement estimate, and a priority queue scoring domains on staleness (0.50), config priority flag (0.35), and anomaly recency (0.15).
Censorship and information control · Engineering and infrastructure
How Voidly selects and maintains the domains it probes for censorship: Citizen Lab's global test list, 12 OONI category codes, per-country supplemental lists, the measurement budget problem, and why the test list is a political document.
Censorship and information control · Engineering and infrastructure
How Voidly protects probe operators in jurisdictions that criminalize censorship measurement: strict data minimization (no name, address, or IP logging), WireGuard peer-key authentication, daily probe ID pseudonymization, optional Tor hidden service upload, measurement scrubbing, country-tier legal risk assessments, and a one-tap emergency stop with full data erasure.
Censorship and information control · Cybersecurity and privacy · Engineering and infrastructure
How a new Voidly probe operator goes from application to publishing measurements: on-device X25519 key generation in the Tauri app, probe registration and ASN verification, 48-hour warmup period with calibration measurements, quality scoring at promotion, and what happens when warmup calibration fails.
Censorship and information control · Engineering and infrastructure
How the Federal Regulatory Data Hub enforces per-client and per-tier rate limits at 8,000 req/s without a centralized counter store: a five-tier quota table (free/researcher/compliance/vendor/internal), token-bucket burst enforcement in Cloudflare KV with ETag-based conditional writes and fail-open after three race retries, and sliding 24-hour window daily quota counting using per-minute KV buckets with a short-lived summary cache for the common below-quota path.
How the Federal Regulatory Data Hub serves 50M+ records via Cloudflare Workers: 8 vertical D1 shards by agency group, Promise.all fan-out for cross-agency queries, entity bridge join across CIK/UEI/LEI/DUNS/NPI, FTS5 full-text search for narrative datasets, response caching with TTL table by endpoint type, and p50/p99 latency budget including partial-response fallback when a shard is unavailable.
How the Federal Regulatory Data Hub implements bitemporal versioning across 50M+ regulatory records in Cloudflare D1: the valid_from/valid_until row-version pattern using half-open intervals, an append-only record_versions audit table with before-state JSON payloads, AS-OF query rewriting in the Workers router using the idx_sdn_pit covering index for sub-5ms p99, three screening modes (current/as-of/historical), and keyset-paginated NDJSON snapshot export for retroactive batch compliance screening.
How the Federal Regulatory Data Hub monitors the freshness of 208 federal datasets and alerts on staleness: per-source FRESHNESS_CONFIG with expected_cadence and max_staleness_hours, D1 dataset_ingests staleness query, Cloudflare Cron */5 * * * * staleness check, multi-channel alerting (Slack webhook, email, PagerDuty) with KV deduplication, OFAC ETag monitoring with 90-minute publish-delay alert, five ingest error classes, and public /status endpoint.
How Voidly selects and distributes its probe vantage network: why ASN diversity matters more than geographic spread, the operator safety constraints that shape high-risk country probes, and how we reach countries where most people connect on mobile-only networks.
Censorship and information control · Engineering and infrastructure
How Voidly delivers measurement configuration to probes without a persistent control channel: gzip+CBOR bundles signed with Ed25519 (signature verified before decompression to prevent zip-bomb attacks), a pull-based auto-update scheduler with 6-hour intervals and exponential backoff, version pinning and two-snapshot rollback, and anonymous country tokens derived via BLAKE3 from ISO code + epoch-week salt so the CDN cannot correlate which overlay a probe applies.
Censorship and information control · Engineering and infrastructure · Cybersecurity and privacy
How Voidly protects probe operator identity while publishing full measurement data: probe_id derived as SHA-256(public_key_bytes) with zero IP logging, human-readable codename system (450K+ combinations, no joint table with probe_id), measurement anonymization (probe_cc + probe_asn published; IP never stored), per-probe Ed25519 signing with isolated key store, and 12-country extra protections (4–48 hour publication delay, 90-day probe_id rotation).
Censorship and information control · Engineering and infrastructure
How the Federal Regulatory Data Hub manages alias proliferation across OFAC SDN, SEC EDGAR, and FinCEN BSA: a five-type alias taxonomy (AKA/FKA/NFE/PHONETIC/VESSEL), entity_aliases DDL with FTS5 virtual table and covering indexes, a normalization pipeline with iterative legal-suffix stripping and NFKD ASCII transliteration, double-Metaphone phonetic bucket generation, and a four-pass resolution pipeline (exact 71.4% → phonetic 88.2% → FTS5 96.1% → edit-distance 98.7% cumulative recall on 2.4M aliases).
Federal data · Engineering and infrastructure · Sanctions and illicit finance
How the Federal Regulatory Data Hub resolves company identity across five incompatible federal identifier schemes: three-pass resolution strategy (exact ID join, alias table lookup, TF-IDF fuzzy name matching), the entity_master bridge table schema, company name normalization to remove legal suffixes, false positive rates by method, special cases for healthcare NPI arrays and foreign entities, and how the entity bridge achieves p50 38ms cross-agency query latency.
The full schema design behind the Federal Regulatory Data Hub: eight vertical D1 databases (securities 9.2M, financial-crimes 4.1M, healthcare 6.8M, labor-safety 3.4M, environment 2.9M, transportation 4.6M, enforcement 2.1M, infrastructure 2.9M), OFAC SDN and EPA enforcement table DDL with FTS5 virtual tables, entity_master bridge with shard_presence bitmask, covering indexes vs. FTS5 trade-offs, and the Workers queryEntityAllShards() Promise.all fan-out achieving p50 38ms cross-shard entity queries.
How the Federal Regulatory Data Hub implements full-text search across 50M+ records using SQLite FTS5 in Cloudflare D1: virtual table creation with the unicode61 tokenizer and content= shadow-table pattern, BM25 scoring with weighted columns (10× entity_name, 5× description, 1× narrative), highlight() and snippet() functions for context extraction, buildFts5Query() TypeScript alias expansion with legal suffix stripping, Promise.all cross-dataset fan-out across 5 D1 shards, trigger-based index maintenance, and weekly optimize via Cloudflare Cron.
How we built a 35M-record federal regulatory database on Cloudflare D1 — per-vertical SQLite tables across 208 datasets, daily cron ingest, FTS5 for free-text datasets, and vertical sharding past the 10GB limit.
How Voidly manages storage for 2.2B probe measurements using a three-tier TimescaleDB retention policy — full-resolution hot tier (0-30 days), native-compressed warm tier (31-365 days, 6.2x ratio), and downsampled cold tier (>365 days, aggregates only), with continuous aggregate cascade, pg_cron compliance verification, and R2 tiered storage planned for Q3 2026.
Censorship and information control · Engineering and infrastructure
How Voidly stores and queries 2.2 billion censorship probe results in TimescaleDB: hypertable design with 1-day chunk intervals and secondary country partitioning, 6.2× compression, continuous aggregates for country-level daily summaries, three-tier retention (hot/warm/cold), and query benchmarks for anomaly detection.
Censorship and information control · Engineering and infrastructure
How Voidly's corroboration engine fetches and aligns data from three independent sources in near-real-time despite their different latency profiles: tokio::join! parallel fetches with per-source timeouts, adaptive OONI polling (15m/60m/3h/6h), in-memory CensoredPlanet daily dump index, independence-weighted source agreement scoring, and retroactive nightly reprocessing against the CP daily dump.
Censorship and information control · Engineering and infrastructure
How the Voidly MCP server exposes 83 tools for querying the global censorship dataset from Claude, GPT, and agent frameworks — incident lookup, measurement queries, country summaries, BGP events, shutdown forecasts, and wiring it into Claude Code.
Censorship and information control · Engineering and infrastructure
How the nightly Voidly export job extracts measurements from TimescaleDB and pushes Parquet snapshots to HuggingFace Hub: PyArrow schema with dictionary-encoded columns, server-side cursor streaming at 50K rows per round-trip, Zstandard level 3 compression, country + year_month partitioning, atomic HuggingFace commit with CommitOperationAdd, post-push SHA-256 verification, and the incremental vs. monthly full-snapshot strategy.
Censorship and information control · Engineering and infrastructure · Transparency and open data
How the Voidly CC BY 4.0 measurement dataset and the OONI historical corpus are hosted on HuggingFace — Parquet snapshot structure, daily incremental updates, git-lfs versioning, and Python/R filter recipes for journalists, ML researchers, and infrastructure teams.
Censorship and information control · Transparency and open data · Engineering and infrastructure
How a Voidly measurement moves through three confidence tiers — Anomaly, Corroborated, Verified Incident — and what each tier means for journalists, ML researchers, and infrastructure monitoring teams using the dataset.
Censorship and information control · Engineering and infrastructure
How the Voidly SSE streaming endpoint delivers censorship events in real time: GET /v1/stream with country/tier/type filtering, four event types (incident_created, incident_updated, incident_resolved, country_status_change), Last-Event-ID reconnection with 24-hour event ring buffer, Python httpx.Client and JavaScript EventSource examples, and how SSE differs from the webhook delivery system.
Censorship and information control · Engineering and infrastructure
How the Voidly REST API is designed: key endpoints for incident lookup, measurement queries, country summaries, domain history, BGP events, and 7-day shutdown forecasts; cursor-based pagination, filtering, rate limits, and code samples in curl, Python, and JavaScript.
Censorship and information control · Engineering and infrastructure
How the Voidly API handles authentication: two access tiers (public 60 req/hr and keyed), voidly_{env}_{base58} key format with PBKDF2-HMAC-SHA256 storage, D1 + KV request authentication flow, four plan tiers (Free/Research/Professional/Enterprise), HMAC-SHA256 webhook signature verification, key rotation without downtime, test keys for CI, and OAuth2 for third-party integrations.
Censorship and information control · Engineering and infrastructure
How Voidly gets verified censorship incidents to journalists, researchers, and monitoring systems: HMAC-signed webhook delivery with exponential-backoff retry, PGP-encrypted email for verified alerts, per-country and per-confidence-tier RSS feeds, alert deduplication by incident_id, and rate-limiting to prevent fatigue.
Censorship and information control · Engineering and infrastructure
How Voidly transitions a censorship incident through five states (Anomaly/MultiSourceAnomaly/Corroborated/Verified/Resolved) with threshold-gated transitions, stores every state change as an append-only event in a TimescaleDB hypertable with SHA-256 idempotency_key, and fans out verified incidents to alert delivery and cache invalidation via three Kafka topics — with the compute_incident_id() Rust function that makes incident IDs deterministic across pipeline restarts.
Censorship and information control · Engineering and infrastructure
How Voidly schedules 80-domain probe runs across 37+ nodes: domain priority tiers by OONI category code, anomaly-driven priority boosts, protocol selection per domain, ±15% jitter for anti-detection, ASN distribution to ensure cross-ASN coverage, adaptive scheduling that injects urgent re-measurements on anomaly detection, and per-country task budgets (CN 68, IR 74, RU 72, global avg 49 tasks/window).
Censorship and information control · Engineering and infrastructure
How the Voidly ML classifier distinguishes DNS tampering, TLS interference, HTTP blocking, BGP withdrawal, and throttling — five per-class binary models, country-specific calibration, and why 95% recall beats 95% precision when cross-source corroboration filters the noise.
Censorship and information control · Machine learning and OSINT · Engineering and infrastructure
How Voidly constructs a labeled training dataset for the anomaly classifier from 200M+ OONI measurements: weak supervision with Snorkel-style label functions across DNS/TCP/TLS/HTTP layers, class imbalance handling with SMOTE and log-weighting, time-based train/val/test splits to prevent leakage, per-country Platt scaling calibration, and the continuous retraining pipeline.
Censorship and information control · Machine learning and OSINT · Engineering and infrastructure
How the quality filter pipeline decides which raw measurements are fit for ML training: boolean checks for control_failure (1.9% drop rate — ISP blocks on control server IPs in CN/IR/RU), missing_fields (0.8%), old probe version pre-2.5.0 (0.3%), and duplicates (0.2%), totalling 3.2% dropped. Includes the quality_filter() Python function, the to_feature_input() schema transformation, and why rejected measurements go to quarantine not discard.
Censorship and information control · Machine learning and OSINT · Engineering and infrastructure
How Voidly normalizes 200M+ OONI measurements across five web_connectivity schema versions (v0.2 to v0.6) into a single ML-ready format: a detect_web_connectivity_version() function using field-presence inference, AnomalyType and ConfidenceTier enums, the OoniMeasurementNormalized dataclass, FLAG_* bitmask constants for DNS/TCP/TLS/HTTP anomaly encoding, side-by-side normalize_v05() vs. normalize_v06() implementations, and a 95.3% pass-through rate from the drop-reason table.
Censorship and information control · Engineering and infrastructure · Machine learning and OSINT
How we processed the OONI raw measurement archive into a flat ML-ready CSV: handling probe version schema drift across 12 years, normalizing test_keys across 20 measurement types, streaming 200M+ records, and what we decided to leave out.
Censorship and information control · Engineering and infrastructure
How Voidly attributes censorship infrastructure to specific DPI vendors using network signatures and open-source intelligence: a six-vendor signature table (TSPU/Sandvine/NetClean/Iran ARRS/Cisco IronPort/GFW), DpiVendorSignature dataclass with a score_signature_match() function weighting RST timing (0.35), block page (0.30), injection IP (0.25), and CA SPKI (0.10), procurement scraping with PROCUREMENT_SOURCES across five government tender portals, BGP TTL-hop attribution, and case studies for Russia, Iran, and Ethiopia.
Censorship and information control · Machine learning and OSINT · Engineering and infrastructure
How we build persistent cross-platform entity profiles for OSINT: passive collection from 40+ sources, graph-based identity disambiguation with calibrated edge weights, Certificate Transparency log monitoring, BGP/ASN change tracking, stylometric fingerprinting, and operational security architecture for researchers in hostile environments.
How the AI Analytics OSINT pipeline extracts, disambiguates, and stores named entity mentions from 58M social media posts per day — GPU-accelerated NER, Wikidata QID linking, cross-language transliteration, and person co-reference resolution.
How we collect and normalize social media data from 47 platforms into a canonical post format: three-tier collection strategy (official APIs, ActivityPub, RSS/scrape), token-bucket rate limiting with circuit breakers, FastText language detection at ingest, content-hash deduplication, and Kafka topic partitioning by platform.
NLP models powering the OSINT platform at 667 posts/second: FastText lid.176 language detection (99.7% EN accuracy), custom SpaCy NER fine-tuned on 2.3M labeled examples across 7 political entity types (91.4% macro F1), DistilBERT fine-tuned on 5M examples with INT8 ONNX quantization (94.7% macro F1, 28ms GPU), MinHash character 4-gram coordinated-campaign detection (89% precision), and the social signal integration with Voidly censorship event detection.
How we detect coordinated amplification campaigns across 58M daily posts: MinHash LSH (128 hash functions, 16 bands, Jaccard threshold 0.80) for content similarity, Redis sorted-set burst detection (≥5 accounts within 15 minutes, inverse-sqrt account age weighting), seven account-feature logistic regression, network amplification ring detection via cycle enumeration, cross-platform timing joins, and a 0–100 coordination score with 70/90 thresholds for human review and auto-flagging.
Machine learning and OSINT · Engineering and infrastructure · Money in politics
How the election intelligence pipeline resolves FEC committee identity across 1.3M records: the 10-code committee type taxonomy (H/S/P/X/Y/N/Q/O/I/U), a JointFundraisingCommittee dataclass with JFCAllocation and resolve_jfc_participants() from Form 99, normalize_entity_name() with iterative legal-suffix stripping, a four-pass resolution table (exact ID 63.4% → exact name 82.1% → alias 91.7% → TF-IDF char 3-gram 95.5% cumulative recall), and LLC chain disambiguation via FinCEN/EDGAR/SOS cross-reference.
How the election intelligence pipeline ingests AP Election API feeds, state authority data (JSON/CSV/HTML scraping), social media signals, and media coverage in real time: Kafka election.precinct_results topic (50 partitions by state FIPS), PrecinctResult protobuf schema, state scraper StateScraperConfig, ElectionSentimentConsumer, narrative divergence scoring, FIPS normalization edge cases (Connecticut planning regions, Alaska districts), and p50/p99 latency targets for all four streams.
Money in politics · Engineering and infrastructure · Machine learning and OSINT
Kafka partition key design, binary COPY writes to TimescaleDB, character 4-gram MinHash LSH distributed across Redis, autoscaling on consumer lag, and a canonical normalization layer across 47 platform schemas — the full pipeline behind 58M posts/day.
All topics: the writing index.