The Trust Store You Never Updated: Why curl, Python, and IoT Didn't Catch eMudhra

The Browser-vs-Everything-Else Gap

Every write-up of the eMudhra incident spends paragraph one on the keys and paragraph two on the fast browser response. Few spend much time on the thing that keeps PKI practitioners up at night: the browsers are a small fraction of the TLS clients that validate certificates every day. The rest of the ecosystem - command-line tools, SDKs, package managers, IoT devices, OT gear, cloud SDK clients, CI/CD runners, service-to-service HTTPS - has, roughly, none of the fast-revocation infrastructure the browsers built.

The browser model works because browsers:

1. Ship on a 2-6 week update cadence
2. Maintain centrally-managed revocation lists (CRLSet, OneCRL, Apple's blocklist) that update out of band
3. Enforce Certificate Transparency (reject certs not in SCTs)
4. Have well-funded root program governance (CCADB, Mozilla, Chrome, Apple)
5. Hard-fail on revocation check for EV, soft-fail for DV but still display warnings

Non-browser clients ship with:

1. OS-provided trust bundle on an OS update cadence (quarterly, if that)
2. Soft-fail on revocation checks (or no revocation checks at all)
3. No CT validation
4. No equivalent of CRLSet / OneCRL
5. Trust-store configuration that is often immutable (compiled into the binary, pinned in firmware)

This post walks through the specific gaps in six classes of non-browser client, and the knobs you can turn today.

1. The curl | sh Problem

The idiomatic install flow for a startling number of developer tools is:

bash
1curl -fsSL https://sh.rustup.rs | sh

What this does:

• curl makes an HTTPS request to sh.rustup.rs
• curl validates the TLS chain against the system trust bundle
• curl streams the response body to sh

What curl does not do by default:

• Check Certificate Transparency
• Check OCSP (unless --cert-status is passed, and even then only soft-fail)
• Check CRLs
• Pin the issuing CA

An adversary who recovered a private key from one of the compromised eMudhra intermediates could, until the OS trust bundle update propagated, serve a forged certificate for sh.rustup.rs from any network position that can intercept the TLS connection. The forged script runs with the user's privileges. For sudo bash | curl setups - which exist - it runs as root.

The specific gap: the attack window is bounded by the OS trust bundle update cadence, not the browser revocation list cadence. For Linux distributions, that's typically weeks. For developers on older distros or custom images, it's longer.

What to turn on:

• curl --pinnedpubkey for installation URLs you control
• A hard-pin of the Sectigo AAA Certificate Services SPKI if your downloads chain through it, or - better - move to Let's Encrypt / ZeroSSL for your public install endpoints and pin the Let's Encrypt roots
• For your own distribution: sign the installer script with a key you control, not the TLS cert

2. Package Managers

Every package manager that resolves transitive dependencies over HTTPS is in the same position:

• npm / yarn / pnpm fetching from registry.npmjs.org
• pip fetching from pypi.org
• cargo fetching from crates.io
• go get fetching from proxy.golang.org and arbitrary github.com paths

None of these check CT. None of them hard-fail on revocation. All of them chain through the system trust bundle, which includes the compromised eMudhra chain for the window between the ceremony and the OS trust-bundle update.

The attack is: forged cert for the registry hostname, package metadata response is intercepted, dependency resolver fetches a malicious version. For a developer with Rust tooling installed, a forged crates.io certificate rewrites Cargo.lock resolution during a build.

What to turn on:

• Package manager content pinning (.pip-tools-compile's hashes, npm shrinkwrap with integrity hashes, Cargo's checksum in Cargo.lock). These check the content of the package after the fact, so even if the TLS connection is spoofed, the hash mismatch catches it.
• Private artifact mirrors terminating TLS at a CA you control.
• Sigstore-backed transparency on package contents - this is now live for npm and pypi, enable it in your CI.

3. Server-to-Server HTTPS

Any service in your stack that calls a third-party API over HTTPS is a candidate for this class of attack:

• Payments to Stripe, Adyen, Braintree
• Webhooks received from GitHub, Linear, Stripe
• OAuth token exchanges
• OIDC JWKS fetches
• Cloud SDK clients to AWS, Azure, GCP
• Database proxies, cache clients
• Internal service mesh if you're not on mTLS

The Go, Python, Node, and Java default http clients all do chain validation against the system trust store. None do CT. None hard-fail revocation. The window of exposure to a stolen eMudhra key is from "ceremony date" to "trust bundle update propagated to every production host".

What to turn on:

• SPKI pinning in your HTTP client for critical outbound endpoints:
  • Go: http.Transport.TLSClientConfig.VerifyPeerCertificate
  • Python: requests with a custom HTTPAdapter and manual SubjectPublicKeyInfo check
  • Node: https.request({ checkServerIdentity })
  • Java: a custom X509TrustManager
• Hard-fail revocation at the proxy layer (Envoy, Linkerd, HAProxy all support this)
• Internal mTLS - this doesn't help against a compromised internal CA, but it does limit the blast radius to internally-issued intermediates you control

4. CI/CD Runners

A GitHub Actions runner, a GitLab Runner, a Buildkite agent, a Jenkins slave - all of these are long-lived machines that:

• Have cloud credentials in environment or OIDC identity federation
• Pull code from github.com / gitlab.com over HTTPS
• Pull dependencies from registry.npmjs.org etc.
• Push artefacts to container registries, S3, etc.
• Often ship with a system trust bundle that is weeks or months out of date

A runner compromised through a forged registry certificate is a runner that can leak production credentials. The scope of this class of attack is quietly enormous.

What to turn on:

• Short-lived cloud credentials via OIDC federation (so a compromised runner can't exfiltrate long-lived keys)
• Ephemeral runners that are destroyed after each job
• Egress proxies that terminate TLS and enforce CT + revocation on behalf of all CI traffic
• Runner image rebuilds on a weekly cadence so the trust bundle stays fresh

5. IoT and OT Devices

The hardest class. The devices are:

• Shipped between 2015-2024
• Rated for 10-15 years of field life
• Have a trust store compiled into firmware
• Have no over-the-air trust-store update path (or a path that was never exercised)
• May be behind a firewall that prevents them from checking OCSP responders
• Are often doing certificate-based mutual auth to a management plane that terminates on a cert chaining through a public root

The specific device-category list that was affected by the eMudhra compromise is: anything that pinned Sectigo's AAA Certificate Services and has no update path. For public utility industrial control gear, connected medical devices, factory-floor PLCs, and connected vehicle telematics, this is a non-trivial fraction of the installed base.

What to do:

• Build the over-the-air trust-store update path in the next device revision, not after the next incident
• For currently-fielded devices without update paths: configure the management plane to pin a specific intermediate (or use a private CA you control for all device communications), reducing the attack surface to intermediates you don't share with the public web
• Document which fielded SKUs trust which anchors - during an incident, this is the inventory that determines who you call first

6. Code Signing

The two code-signing intermediates among the eMudhra batch are a specific hazard. Windows code-signing policy:

• Checks the chain against the OS trust store
• Checks for timestamping (so the signature survives cert expiry)
• Soft-fails on revocation checks by default - an unreachable OCSP responder results in the signature being considered valid

An adversary with a key from the eMudhra code-signing intermediate could sign a Windows binary that would appear legitimately signed to any Windows system until the Windows trust bundle updates to revoke the intermediate. Timestamping the signature extends this past the original cert's revocation.

The macOS Gatekeeper story is better - Apple's own notarization service is a parallel check - but any binary signed before Apple's revocation reaches a given machine is validated against the then-current trust bundle.

What to turn on:

• Hard-fail revocation in your SOC monitoring for anything signed by an intermediate chaining through Sectigo AAA
• Sigstore-backed transparency for all internal code signing (so a forged signature from a public CA doesn't automatically count as "ours")
• For publicly-distributed binaries, migrate to EV code signing with HSM-backed keys from a CA you have direct contractual relationship with

The Inventory Problem

The through-line of every one of these mitigations is the same: you need to know what your systems actually trust before you can know what you're exposed to.

Most production estates do not have this inventory. The trust bundle on a 3-year-old Ubuntu image is not the same as the one on a current one. The trust bundle compiled into a Go binary (via x509.SystemCertPool()) depends on when the binary was built and what base image it was built against. The trust bundle on a Fortinet firewall was last updated when the operator thought about it. The trust bundle on an IoT device shipped in 2019 is frozen.

For TigerTrust customers, this is what the agent trust-store enumeration surfaces - every anchor currently present on every host, every anchor the host is actively using to validate chains, cross-referenced against known-good lists. In the specific case of the eMudhra incident, operators can grep their estate for "any host that trusts the Sectigo AAA Certificate Services root" and the answer comes back in minutes, not weeks.

For non-TigerTrust estates: build the inventory out of openssl crl2pkcs7 -nocrl -certfile /etc/ssl/certs/ca-certificates.crt | openssl pkcs7 -print_certs running under configuration management, parse the subject lines, store the per-host fingerprint lists in a central inventory. It is not clever, but it is the inventory you need the next time this happens.

The Underlying Point

The browser response to the eMudhra incident was fast and well-governed. The non-browser response was neither of those things - not because the non-browser ecosystem is poorly operated, but because the governance machinery that enables fast browser response doesn't extend outside the browsers.

The pragmatic shape of the fix is layered:

1. Inventory what you trust (do this before the next incident)
2. Pin critical paths (so trust-store changes don't move you)
3. Hard-fail revocation where you can
4. Shorten everything's lifetime (so stolen keys age out faster)
5. Build the update path for anything that will outlive the current CA cohort

None of these are new. All of them are more urgent after eMudhra than before.

Related Reading

• eMudhra, eight bytes of entropy, and what breaks - the incident analysis
• The multi-CA governance model after eMudhra - what this incident means for root programs
• Scott Helme: Eight Bits of Trust - the original write-up