Repository navigation
fix(express): rebuild image to drop eMudhra TLS trust - #10021
ralph-bitgo[bot] wants to merge 1 commit into
Conversation
bitgo/express:17.6.0 validates TLS against Node's embedded root store, which still trusts the eMudhra roots compromised in the October 2026 CA incident, and ships no system CA bundle, so TLS-sensitive workloads behind Express accept eMudhra-rooted chains. - Move the base to node:24.15.0-bookworm-slim (newest available base; .nvmrc already tracks Node 24). - Install ca-certificates so a usable system bundle exists, and strip the three eMudhra roots Debian 12 still ships until the distro removes them. - Set NODE_OPTIONS=--use-openssl-ca and SSL_CERT_FILE so outbound TLS validates against that bundle instead of Node's embedded roots. - Add scripts/verify-express-image-trust.sh and run it in the CI docker-build job. It asserts the bundle exists and is free of eMudhra trust and that TLS verification follows the system store. Corrected Node trust data remains outstanding: v24.15.0's embedded root store still lists emSign Root CA - G1/C1 and emSign ECC Root CA - G3/C3. --use-openssl-ca keeps that store out of TLS verification, but a base bump to a Node release with corrected trust data is still needed once a suitable tag is available. Ticket: COINS-1861 Session-Id: f02afc86-4d81-4a81-99b8-96565f3c39ac Task-Id: e88e5f0c-081b-4523-b980-8235f9aefc3a Requested-By: Otto Allmendinger <otto@bitgo.com>
527cd49 to
d0fb09f
Compare
![ralph-bitgo[bot]](https://avatars.githubusercontent.com/in/4074619?s=60&v=4){kind=small}
| # https://stackoverflow.com/a/63142468/134409 | ||
| # https://medium.com/@emilefugulin/building-a-sane-docker-image-for-typescript-lerna-and-prisma-2-76d8ff9926e4 | ||
| FROM node:22.22.0-bookworm-slim@sha256:f86be15afa9a8277608e141ce2a8aa55d3d9c40845921b8511f4fb7897be2554 AS filter-packages-json | ||
| FROM node:24.15.0-bookworm-slim@sha256:4e6b70dd6cbfc88c8157ba19aa3d9f9cce6ba4703576d55459e45efcbc9c5f5d AS filter-packages-json |
There was a problem hiding this comment.
The digest is the upstream Docker Hub RepoDigest for node:24.15.0-bookworm-slim as recorded in BitGo/docker-hub-sync's mirror list (the ECR mirror digest differs). Heads-up: this base's embedded root store still lists the eMudhra roots — v24.15.0 does not carry the corrected trust data of Node >= 22.23.4, which is why verification is moved to the system store and that requirement is called out as outstanding in the body.
| RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates tini && rm -rf /var/lib/apt/lists/* && \ | ||
| for f in $(find /usr/share/ca-certificates -name '*.crt'); do \ | ||
| if openssl x509 -in "$f" -noout -subject 2>/dev/null | grep -qi 'emudhra\|emsign'; then \ | ||
| echo "removing compromised trust anchor: $f" && rm "$f"; \ | ||
| fi; \ | ||
| done && \ | ||
| sed -i -e '/emSign/d' -e '/emudhra/Id' /etc/ca-certificates.conf && \ | ||
| update-ca-certificates --fresh |
There was a problem hiding this comment.
Debian 12's ca-certificates (20250419~deb12u2) still ships three eMudhra roots, so installing the package alone would swap one eMudhra trust path for another. This scrub is temporary and idempotent: once the distro drops the roots the loop and sed match nothing, and the verify script asserts the end state either way.
| # Validate outbound TLS against the system CA bundle instead of Node's embedded | ||
| # root store, so OS trust updates (e.g. revoked roots) take effect. | ||
| ENV NODE_OPTIONS=--use-openssl-ca | ||
| ENV SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt |
There was a problem hiding this comment.
NODE_EXTRA_CA_CERTS can only add anchors to the trusted set — it cannot remove the eMudhra roots from Node's embedded store. So the remediation moves verification off that store entirely: in Node's NewRootCertStore() --use-openssl-ca selects X509_STORE_set_default_paths as a mutually exclusive alternative to the bundled roots, and SSL_CERT_FILE pins those paths to the scrubbed bundle regardless of the compiled-in OPENSSLDIR.
| } | ||
| ' | ||
|
|
||
| # TLS verification must follow the configured system bundle. A server whose |
There was a problem hiding this comment.
The accept/reject pair is the real assertion: accept proves verification follows the configured bundle (dropping --use-openssl-ca breaks it, since the bundled roots then reject the probe CA), and reject proves the default bundle grants no trust by accident (removing the scrub or replacing the config with NODE_EXTRA_CA_CERTS alone breaks it). The embedded-store line above is informational because that store is not loaded under this configuration.
| GIT_HASH=${{ github.sha }} | ||
| SOCKET_SECURITY_MODE=${{ env.SOCKET_SECURITY_MODE }} | ||
|
|
||
| - name: Verify CA trust configuration |
There was a problem hiding this comment.
docker-build is the only gate that builds the real image from the Dockerfile, so the trust check lives here — it re-verifies every future base-image bump too, which is exactly how stale embedded trust data in a new base would get caught.
3 participants
Description
The public
bitgo/express:17.6.0image (Node v22.22.0) validates outbound TLS against Node's embedded root store, which trusts the eMudhra roots compromised in the October 2026 CA incident, and the image contains no system CA bundle at all. TLS-sensitive workloads running behind BitGo Express therefore accept eMudhra-rooted chains (COINS-1861 reproduction:ACCEPTED HTTP=200). This PR rebuilds the image so its effective TLS trust excludes eMudhra.What changed
Dockerfile: base bumpednode:22.22.0-bookworm-slim→node:24.15.0-bookworm-slim(digest-pinned; digest taken from the BitGo/docker-hub-sync mirror list, which records upstream Docker Hub digests). The runtime stage now installsca-certificatesand strips the three eMudhra roots Debian 12 still ships (ca-certificates20250419~deb12u2:emSign Root CA - G1,emSign Root CA - C1,emSign ECC Root CA - C3). Runtime env setsNODE_OPTIONS=--use-openssl-caandSSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crtso TLS verification follows the system bundle instead of Node's embedded roots.scripts/verify-express-image-trust.sh(new), run by the CIdocker-buildjob: asserts the bundle exists and carries no eMudhra/emSign trust, and behaviorally that a chain anchored in a throwaway probe CA verifies only when that CA is in the configured bundle — i.e. the system store is the verification path and nothing else grants trust.modules/express/README.md: documents the TLS trust configuration and how to trust an additional CA.Remediation status vs the requested items
emSign Root CA - G1/C1andemSign ECC Root CA - G3/C3(X.509-parsed in the built image; the pre-22.22.0tls.rootCertificatesdata is not corrected here). Under--use-openssl-cathat store is not consulted for TLS verification, so the acceptance behavior is remediated and asserted by CI, but the embedded trust data itself is not corrected. A follow-up base bump is required once a Node release with corrected trust data (22.23.4-equivalent) is published and mirrored — no such tag exists in the BitGo ECR mirror today (docker.io/library/nodemirrors only 18.x, 20.x, and 24.15.0; Docker Hub is not reachable from agent environments to pin an unmirrored tag).ca-certificates+ usable system bundle (/etc/ssl/certs/ca-certificates.crt, 145 certs after scrub).NODE_OPTIONS=--use-openssl-ca+SSL_CERT_FILE).npmjs-release.ymlrelease workflow builds and pushesbitgo/express:<version>and re-pointslatest, producing the new immutable digest. After merge, run the release workflow to ship the rebuilt image. Until then the currentlatestshould not be used for TLS-sensitive workloads.Issue Number
COINS-1861 — https://linear.app/bitgo/issue/COINS-1861/rebuild-bitgo-express-image-to-remove-compromised-emudhra-trust
Type of change
How Has This Been Tested?
Dockerfilelines (base + CA install/scrub + runtime env) and ranscripts/verify-express-image-trust.shagainst it: system bundle 145 certs / 0 eMudhra roots; probe-CA chainACCEPTEDwith the probe bundle andREJECTED UNABLE_TO_VERIFY_LEAF_SIGNATUREwith the default bundle. This proves TLS verification follows the configured system store and that store trusts no eMudhra, and it would fail if--use-openssl-cawere dropped, the scrub were removed, orNODE_EXTRA_CA_CERTSwere substituted.--use-openssl-ca, marked outstanding above).docker-buildjob on this PR builds the full image, runs the same verification script, and hits/api/v2/ping.yarn install→lerna run buildstep was OOM-killed three times (exit 137) on a memory-pressured host. That step is unchanged by this diff; the CI docker-build job is the full-build verification. Unit tests: N/A (no TypeScript changes).Checklist:
Ticket: COINS-1861