Voidly probe config delivery: signed bundles, auto-update protocol, and country-specific measurement parameters

A network measurement probe cannot maintain a persistent control channel. The measurement act itself is the probe's primary observable network behaviour, and a live TCP connection to a known control server would be trivially blockable — or worse, would reveal the probe operator's identity to a surveillance middlebox. Voidly probes therefore operate on a pull model: at startup and on a configurable schedule, the probe fetches a signed configuration bundle from a CDN-backed URL and applies it locally. No persistent connection, no server-side knowledge of which probe fetched what.

This article covers the config bundle format, the Ed25519 signing and verification protocol, the pull-based auto-update scheduler with version pinning and rollback, and the country-specific parameter override system that allows Voidly to send Iran probes a different measurement cadence than probes in Germany without the CDN or any intermediate party learning which country the probe is in.

Config bundle format

A config bundle is a gzip-compressed CBOR document followed by a detached Ed25519 signature. The probe fetches the bundle and its companion .sig file, verifies the signature against a set of trusted public keys baked into the probe binary, and then parses the CBOR payload. CBOR was chosen over JSON to minimize bundle size on cellular connections (typical bundle: 18 KB gzip vs. 64 KB JSON) and to allow strict schema validation without a separate schema document.

The Rust struct definitions for the config bundle:

// src/config/bundle.rs

use serde::{Deserialize, Serialize};
use std::collections::HashMap;

/// Top-level config bundle — deserialized from CBOR after signature verification.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ConfigBundle {
    /// Monotonically increasing integer; probes reject bundles with version <= current.
    pub version: u64,

    /// ISO 8601 datetime; probes reject bundles older than 72 hours.
    pub issued_at: String,

    /// Measurement schedule common to all probes.
    pub global: GlobalConfig,

    /// Country-specific overrides keyed by anonymous country token (not ISO code).
    /// The probe derives its own token; no external lookup required.
    pub country_overrides: HashMap<String, CountryConfig>,

    /// Test list metadata: version hash, fetch URL template, expected item count.
    pub test_list: TestListRef,

    /// Classifier model metadata: version, fetch URL, expected SHA-256.
    pub model_ref: ModelRef,
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct GlobalConfig {
    /// Seconds between measurement cycles when on WiFi.
    pub cycle_interval_wifi_secs: u32,

    /// Seconds between measurement cycles when on cellular.
    pub cycle_interval_cellular_secs: u32,

    /// Maximum domains measured per cycle.
    pub domains_per_cycle: u16,

    /// Maximum cumulative upload bytes per 24h window (across all measurements).
    pub daily_upload_cap_bytes: u64,

    /// Minimum battery percentage before skipping a cycle (0 = no limit).
    pub battery_floor_pct: u8,

    /// HTTP measurement timeout in milliseconds.
    pub http_timeout_ms: u32,

    /// DNS measurement timeout in milliseconds.
    pub dns_timeout_ms: u32,

    /// TLS measurement timeout in milliseconds.
    pub tls_timeout_ms: u32,
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CountryConfig {
    /// Overrides global cycle_interval_wifi_secs for this country token.
    pub cycle_interval_wifi_secs: Option<u32>,

    /// Overrides domains_per_cycle for this country token.
    pub domains_per_cycle: Option<u16>,

    /// Additional domain slugs appended to the global test list for this token.
    pub priority_domains: Vec<String>,

    /// Measurement types enabled: "web_connectivity" | "dns_consistency" | "tcp_connect"
    pub enabled_test_types: Vec<String>,
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TestListRef {
    pub version_hash: String,      // SHA-256 of the full test list file
    pub fetch_url_template: String, // e.g. "https://cdn.voidly.net/lists/{version_hash}.gz"
    pub item_count: u32,
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ModelRef {
    pub version: String,           // semver
    pub fetch_url: String,
    pub sha256: String,
    pub size_bytes: u64,
}

The country_overrides map uses an anonymous country token rather than an ISO 3166-1 code. The token is a BLAKE3 hash of the probe's country ISO code concatenated with a epoch-week salt published in the global config. The CDN and any network observer therefore see the probe fetching a bundle that contains a map of opaque tokens — they cannot determine which token the probe selected unless they also know the salt-to-country mapping, which is never published. The probe applies only the entry that matches its own derived token and ignores all others.

Ed25519 signing and verification

Voidly maintains three Ed25519 signing keys in a hardware security module. Two keys are active simultaneously (current and pending) to allow zero-downtime key rotation; the third is a recovery key stored offline. The probe binary embeds all three corresponding public keys. A config bundle is accepted if any one of the three embedded public keys verifies the signature:

// src/config/verify.rs

use ed25519_dalek::{Signature, VerifyingKey, Verifier};

/// Public keys baked into the probe binary at build time.
/// Key rotation adds a new key here; old key remains until all probes have updated.
pub const TRUSTED_VERIFYING_KEYS: &[&[u8; 32]] = &[
    b"\x9b\x2a...32 bytes...",  // current key (rotated quarterly)
    b"\x4f\x17...32 bytes...",  // pending key (becomes current at next rotation)
    b"\x01\xe3...32 bytes...",  // recovery key (used only if current+pending both compromised)
];

pub fn verify_bundle(bundle_bytes: &[u8], sig_bytes: &[u8]) -> Result<(), ConfigError> {
    let signature = Signature::from_slice(sig_bytes)
        .map_err(|_| ConfigError::InvalidSignatureFormat)?;

    for raw_key in TRUSTED_VERIFYING_KEYS {
        let vk = VerifyingKey::from_bytes(raw_key)
            .map_err(|_| ConfigError::MalformedPublicKey)?;
        if vk.verify(bundle_bytes, &signature).is_ok() {
            return Ok(());
        }
    }

    Err(ConfigError::SignatureVerificationFailed)
}

pub fn load_and_verify_bundle(
    bundle_bytes: &[u8],
    sig_bytes: &[u8],
) -> Result<ConfigBundle, ConfigError> {
    // Signature check BEFORE decompression to avoid zip-bomb attacks
    verify_bundle(bundle_bytes, sig_bytes)?;

    // Decompress (gzip)
    let decompressed = decompress_gzip(bundle_bytes)?;

    // Check decompressed size bound (max 2 MB after decompression)
    if decompressed.len() > 2 * 1024 * 1024 {
        return Err(ConfigError::BundleTooLarge(decompressed.len()));
    }

    // Deserialize CBOR
    let bundle: ConfigBundle = ciborium::de::from_reader(decompressed.as_slice())
        .map_err(|e| ConfigError::CborDeserializationFailed(e.to_string()))?;

    // Reject stale bundles (older than 72 hours)
    validate_bundle_freshness(&bundle)?;

    Ok(bundle)
}

Signature verification precedes decompression intentionally. A malicious CDN or man-in-the-middle cannot force the probe to decompress an unsigned payload, preventing zip-bomb attacks. The 72-hour freshness window means that even if an attacker replays a legitimately signed old bundle, the probe will reject it within three days.

Pull-based auto-update protocol

The probe's config update scheduler runs independently of the measurement loop. It maintains a local state file tracking the current bundle version, the next scheduled fetch time, and a backoff counter for failed fetches:

// src/config/updater.rs

use std::time::{Duration, SystemTime};

const BASE_RETRY_DELAY_SECS: u64 = 300;    // 5 minutes
const MAX_RETRY_DELAY_SECS:  u64 = 86_400; // 24 hours
const BUNDLE_URL: &str =
    "https://cdn.voidly.net/config/latest.cbor.gz";
const BUNDLE_SIG_URL: &str =
    "https://cdn.voidly.net/config/latest.cbor.gz.sig";

pub struct ConfigUpdater {
    current_version:    u64,
    next_fetch_at:      SystemTime,
    consecutive_errors: u32,
    pinned_version:     Option<u64>,  // set by --pin-config CLI flag
}

impl ConfigUpdater {
    pub async fn tick(&mut self) -> Result<Option<ConfigBundle>, ConfigError> {
        if SystemTime::now() < self.next_fetch_at {
            return Ok(None);
        }

        // Fetch bundle and signature concurrently
        let (bundle_bytes, sig_bytes) = tokio::try_join!(
            fetch_bytes(BUNDLE_URL),
            fetch_bytes(BUNDLE_SIG_URL),
        )?;

        let bundle = load_and_verify_bundle(&bundle_bytes, &sig_bytes)
            .map_err(|e| {
                self.record_error();
                e
            })?;

        // Version pinning: reject bundles with version > pinned_version
        if let Some(pin) = self.pinned_version {
            if bundle.version > pin {
                log::info!(
                    "Bundle version {} exceeds pinned {}; ignoring update",
                    bundle.version, pin
                );
                self.schedule_next_fetch_success();
                return Ok(None);
            }
        }

        // Reject downgrade attempts
        if bundle.version <= self.current_version {
            log::debug!("Bundle version {} not newer than {}; skipping", bundle.version, self.current_version);
            self.schedule_next_fetch_success();
            return Ok(None);
        }

        self.current_version = bundle.version;
        self.consecutive_errors = 0;
        self.schedule_next_fetch_success();

        Ok(Some(bundle))
    }

    fn record_error(&mut self) {
        self.consecutive_errors += 1;
        let delay = BASE_RETRY_DELAY_SECS
            .saturating_mul(1 << self.consecutive_errors.min(7))
            .min(MAX_RETRY_DELAY_SECS);
        self.next_fetch_at = SystemTime::now() + Duration::from_secs(delay);
    }

    fn schedule_next_fetch_success(&mut self) {
        // Re-fetch every 6 hours with ±30 min jitter to avoid thundering herd
        let jitter_secs = (rand::random::<u32>() % 3600).saturating_sub(1800);
        self.next_fetch_at = SystemTime::now()
            + Duration::from_secs(6 * 3600 + jitter_secs as u64);
    }
}

The 6-hour fetch interval means the CDN receives at most ~27M requests per 24-hour period from 40,000 probes. Requests are served from Cloudflare's edge cache with a cache-control TTL of 4 hours on the bundle and 30 seconds on the .sig file. The signature file has a shorter TTL so that a revoked bundle's signature can be invalidated quickly without purging the cached bundle bytes.

Rollback mechanism

If a new config bundle causes a probe crash on startup (detected by a watchdog that monitors the measurement loop heartbeat), the probe's supervisor process restores the previous bundle from a local snapshot. Snapshots are kept for the two most recent successful bundle versions:

// src/config/snapshot.rs

pub struct SnapshotStore {
    store_dir: PathBuf,
}

impl SnapshotStore {
    /// Save a successfully applied bundle for rollback purposes.
    pub fn save(&self, version: u64, bundle_bytes: &[u8], sig_bytes: &[u8]) -> io::Result<()> {
        let bundle_path = self.store_dir.join(format!("{version}.cbor.gz"));
        let sig_path    = self.store_dir.join(format!("{version}.cbor.gz.sig"));
        fs::write(&bundle_path, bundle_bytes)?;
        fs::write(&sig_path,    sig_bytes)?;

        // Prune snapshots older than the two most recent versions
        self.prune_old_snapshots(2)
    }

    pub fn rollback(&self) -> io::Result<ConfigBundle> {
        // Load the second-most-recent snapshot (most recent failed)
        let mut versions = self.list_snapshot_versions()?;
        versions.sort_unstable_by(|a, b| b.cmp(a)); // descending

        let target_version = versions.get(1)
            .copied()
            .ok_or_else(|| io::Error::new(io::ErrorKind::NotFound, "no rollback snapshot"))?;

        let bundle_bytes = fs::read(
            self.store_dir.join(format!("{target_version}.cbor.gz"))
        )?;
        let sig_bytes = fs::read(
            self.store_dir.join(format!("{target_version}.cbor.gz.sig"))
        )?;

        // Re-verify even for local snapshot (defense in depth)
        load_and_verify_bundle(&bundle_bytes, &sig_bytes)
            .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e.to_string()))
    }
}

Country-specific parameter application

At startup, after the bundle passes signature and freshness checks, the probe derives its country token and merges the matching CountryConfig overlay onto the global defaults. Fields present in the overlay replace the global value; absent fields inherit the global default. The merge is performed once at startup and the resulting ResolvedConfig struct is passed to the measurement loop:

// src/config/resolver.rs

pub fn resolve_config(bundle: &ConfigBundle, country_iso: &str) -> ResolvedConfig {
    let week_salt = &bundle.global.country_token_salt;
    let token = derive_country_token(country_iso, week_salt);

    let overlay = bundle.country_overrides.get(&token);

    ResolvedConfig {
        cycle_interval_wifi_secs: overlay
            .and_then(|o| o.cycle_interval_wifi_secs)
            .unwrap_or(bundle.global.cycle_interval_wifi_secs),

        cycle_interval_cellular_secs: bundle.global.cycle_interval_cellular_secs,

        domains_per_cycle: overlay
            .and_then(|o| o.domains_per_cycle)
            .unwrap_or(bundle.global.domains_per_cycle),

        daily_upload_cap_bytes: bundle.global.daily_upload_cap_bytes,
        battery_floor_pct:      bundle.global.battery_floor_pct,
        http_timeout_ms:        bundle.global.http_timeout_ms,
        dns_timeout_ms:         bundle.global.dns_timeout_ms,
        tls_timeout_ms:         bundle.global.tls_timeout_ms,

        priority_domains: overlay
            .map(|o| o.priority_domains.clone())
            .unwrap_or_default(),

        enabled_test_types: overlay
            .map(|o| o.enabled_test_types.clone())
            .unwrap_or_else(|| vec![
                "web_connectivity".to_string(),
                "dns_consistency".to_string(),
                "tcp_connect".to_string(),
            ]),
    }
}

fn derive_country_token(country_iso: &str, salt: &str) -> String {
    use blake3::Hasher;
    let mut h = Hasher::new();
    h.update(country_iso.as_bytes());
    h.update(b"\x00");
    h.update(salt.as_bytes());
    h.finalize().to_hex()[..16].to_string()
}

The epoch-week salt in the bundle rotates every seven days. After rotation, the country token a probe derives for “IR” changes, breaking any attempt by a CDN or intermediary to correlate fetches across weeks even if they somehow learned which token maps to Iran. The new salt is included in the bundle one week before it becomes active, giving all probes time to pre-compute and cache the next week's token before the switchover.

CDN distribution architecture

Config bundles are published to an S3-compatible origin and cached at Cloudflare's edge. The publish pipeline runs in CI after each config change is reviewed and merged:

Versioned bundle objects are never deleted; they provide the rollback material if the live bundle is found to cause probe crashes at scale. A canary publish path pushes a new bundle to acanary.cbor.gz URL first; probes enrolled in the canary channel (approximately 2% of the fleet, randomly selected at commissioning time) fetch from the canary URL and their telemetry is monitored for 4 hours before the bundle is promoted to latest.

Related writing

Voidly probe operator privacy covers the pseudonymous credential system that protects probe operators — the same threat model that motivates the anonymous country token design described above.

Voidly vantage point selection describes how config bundles' country-specific domain priority lists interact with the vantage selection algorithm that decides which probes measure which domains.