Repository navigation
fix(express): rebuild image to drop eMudhra TLS trust #10035
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -35,7 +35,19 @@ RUN \ | |
|
|
||
|
|
||
| FROM node:22.23.4-bookworm-slim@sha256:de170b39439eb0e5a3444ad7d60b015614a381e10fecae2e30ec9e735122f38d | ||
| RUN apt-get update && apt-get install -y tini | ||
| # The runtime validates TLS against the system CA bundle (NODE_OPTIONS at the | ||
| # bottom), so ca-certificates must be installed. Debian's package still trusts | ||
| # eMudhra roots after the October 2026 CA compromise, so they are stripped here | ||
| # until the distro removes them; scripts/verify-express-image-trust.sh asserts | ||
| # that none remain. | ||
| RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates tini && rm -rf /var/lib/apt/lists/* && \ | ||
| for f in $(find /usr/share/ca-certificates -name '*.crt'); do \ | ||
|
Comment on lines
+39
to
+44
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Debian 12's ca-certificates (20250419~deb12u2) still ships eMudhra/emSign roots, so installing the package alone would swap one eMudhra trust path for another. This scrub is temporary and idempotent: once the distro drops the roots the loop and sed match nothing, and the verify script asserts the end state either way. |
||
| if openssl x509 -in "$f" -noout -subject 2>/dev/null | grep -qi 'emudhra\|emsign'; then \ | ||
| echo "removing compromised trust anchor: $f" && rm "$f"; \ | ||
| fi; \ | ||
| done && \ | ||
| sed -i -e '/emSign/d' -e '/emudhra/Id' /etc/ca-certificates.conf && \ | ||
| update-ca-certificates --fresh | ||
| # copy the root node_modules to the bitgo-express parent node_modules | ||
| COPY --from=builder /tmp/bitgo/node_modules /var/node_modules/ | ||
| COPY --from=builder /tmp/bitgo/modules/express /var/bitgo-express/ | ||
|
|
@@ -374,5 +386,9 @@ LABEL git_hash=${GIT_HASH} | |
| USER node | ||
| ENV NODE_ENV=production | ||
| ENV BITGO_BIND=0.0.0.0 | ||
| # Validate outbound TLS against the system CA bundle instead of Node's embedded | ||
| # root store, so OS trust updates (e.g. revoked roots) take effect. | ||
| ENV NODE_OPTIONS=--use-openssl-ca | ||
|
Comment on lines
+389
to
+391
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. NODE_EXTRA_CA_CERTS can only add anchors to the trusted set — it cannot remove compromised roots. So verification moves off Node's embedded store entirely: in Node's NewRootCertStore() --use-openssl-ca selects X509_STORE_set_default_paths as a mutually exclusive alternative to the bundled roots, and SSL_CERT_FILE pins those paths to the scrubbed bundle. That also means future OS trust updates take effect without a Node bump. |
||
| ENV SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt | ||
| EXPOSE 3080 | ||
| ENTRYPOINT ["/usr/bin/tini", "--", "/usr/local/bin/node", "/var/bitgo-express/bin/bitgo-express"] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -97,6 +97,18 @@ Environment: prod | |
| Base URI: https://0.0.0.0:4000 | ||
| ``` | ||
|
|
||
| ### TLS trust | ||
|
|
||
| The BitGo Express image validates outbound TLS against the system CA | ||
| store (`/etc/ssl/certs/ca-certificates.crt`, from the Debian | ||
| `ca-certificates` package) instead of Node's embedded root store. The | ||
| container sets `NODE_OPTIONS=--use-openssl-ca` and | ||
| `SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt`. To trust an | ||
| additional CA (for example an internal CA), set `NODE_EXTRA_CA_CERTS` | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Wording matters here and I verified it empirically against the built image: NODE_EXTRA_CA_CERTS extends the configured store (it does not replace it), while pointing SSL_CERT_FILE at another bundle replaces the default one. |
||
| to a PEM file that contains it — extra certificates are added on top | ||
| of the configured system store — or point `SSL_CERT_FILE` at a bundle | ||
| that includes it (this replaces the default bundle). | ||
|
|
||
| ### Building the Docker container | ||
|
|
||
| To build the BitGo Express Docker container yourself from the source code, check out the latest **master** branch and run `docker build` from the project root. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,114 @@ | ||
| #!/bin/bash | ||
|
|
||
| # Verify a BitGo Express image cannot validate TLS chains rooted in eMudhra | ||
| # (October 2026 CA compromise): the system CA bundle carries no eMudhra/emSign | ||
| # trust, and TLS verification follows that bundle rather than Node's embedded | ||
| # root store. | ||
| # | ||
| # Usage: scripts/verify-express-image-trust.sh <image> | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| IMAGE="${1:?usage: verify-express-image-trust.sh <image>}" | ||
|
|
||
| WORK_DIR=$(mktemp -d) | ||
| trap 'rm -rf "$WORK_DIR"' EXIT | ||
|
|
||
| echo "Verifying CA trust configuration of $IMAGE" | ||
|
|
||
| # The system CA bundle must exist and contain certificates. | ||
| if ! docker run --rm --entrypoint /bin/sh "$IMAGE" -c \ | ||
| 'test -s /etc/ssl/certs/ca-certificates.crt && grep -q "BEGIN CERTIFICATE" /etc/ssl/certs/ca-certificates.crt'; then | ||
| echo "::error::missing or empty /etc/ssl/certs/ca-certificates.crt in $IMAGE" | ||
| exit 1 | ||
| fi | ||
|
|
||
| # The system bundle must carry no eMudhra/emSign trust. Node's embedded root | ||
| # store is reported but only matters when the runtime consults it, which the | ||
| # probes below assert it does not. | ||
| docker run --rm --entrypoint /usr/local/bin/node "$IMAGE" -e ' | ||
| const fs = require("fs"); | ||
| const tls = require("tls"); | ||
| const { X509Certificate } = require("crypto"); | ||
| const pem = fs.readFileSync("/etc/ssl/certs/ca-certificates.crt", "utf8"); | ||
| const blocks = pem.match(/-----BEGIN CERTIFICATE-----[\s\S]*?-----END CERTIFICATE-----/g) || []; | ||
| const compromised = (x509) => /emudhra|emsign/i.test(`${x509.subject}\n${x509.issuer}`); | ||
| const embedded = tls.rootCertificates.map((block) => new X509Certificate(block)); | ||
| console.log(`system bundle certs: ${blocks.length}, embedded roots: ${embedded.length}`); | ||
| const hits = blocks.map((block) => new X509Certificate(block)).filter(compromised); | ||
| if (hits.length > 0) { | ||
| console.error(`eMudhra/emSign trust found in the system bundle:\n${hits.map((x509) => x509.subject).join("\n")}`); | ||
| process.exit(1); | ||
| } | ||
| const embeddedHits = embedded.filter(compromised); | ||
| if (embeddedHits.length > 0) { | ||
| console.log(`note: Node embedded root store lists ${embeddedHits.length} eMudhra/emSign root(s); NODE_OPTIONS=--use-openssl-ca keeps them out of TLS verification (asserted by the probes below)`); | ||
| } | ||
| ' | ||
|
|
||
| # TLS verification must follow the configured system bundle. A server whose | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The accept/reject pair is the real assertion: accept proves verification follows the configured bundle (dropping --use-openssl-ca breaks it, since the bundled roots then reject the probe CA), and reject proves the default bundle grants no trust by accident (removing the scrub or replacing the config with NODE_EXTRA_CA_CERTS alone breaks it). The embedded-store line is informational because that store is never consulted under the configured flags. |
||
| # chain is anchored in a throwaway probe CA is accepted only when that CA is in | ||
| # the bundle Node validates against; Node's embedded roots alone reject it. | ||
| openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \ | ||
| -keyout "$WORK_DIR/probe-ca.key" -out "$WORK_DIR/probe-ca.crt" -days 1 \ | ||
| -subj "/CN=Express Trust Probe CA" | ||
| openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \ | ||
| -keyout "$WORK_DIR/probe-leaf.key" -out "$WORK_DIR/probe-leaf.csr" \ | ||
| -subj "/CN=localhost" -addext "subjectAltName=DNS:localhost,IP:127.0.0.1" | ||
| openssl x509 -req -in "$WORK_DIR/probe-leaf.csr" \ | ||
| -CA "$WORK_DIR/probe-ca.crt" -CAkey "$WORK_DIR/probe-ca.key" -CAcreateserial \ | ||
| -out "$WORK_DIR/probe-leaf.crt" -days 1 -copy_extensions copy | ||
|
|
||
| PROBE_JS=' | ||
| const fs = require("fs"); | ||
| const tls = require("tls"); | ||
| if (process.env.PROBE_CA) { | ||
| fs.writeFileSync("/tmp/probe-bundle.pem", process.env.PROBE_CA); | ||
| } | ||
| const server = tls.createServer( | ||
| { key: process.env.PROBE_KEY, cert: process.env.PROBE_LEAF }, | ||
| (socket) => socket.end("ok") | ||
| ); | ||
| server.listen(0, "127.0.0.1", () => { | ||
| const socket = tls.connect( | ||
| { host: "127.0.0.1", port: server.address().port, servername: "localhost", rejectUnauthorized: true }, | ||
| () => { | ||
| console.log("ACCEPTED"); | ||
| socket.end(); | ||
| server.close(); | ||
| } | ||
| ); | ||
| socket.on("error", (err) => { | ||
| console.log(`REJECTED ${err.code || err.message}`); | ||
| server.close(); | ||
| process.exitCode = 2; | ||
| }); | ||
| }); | ||
| ' | ||
|
|
||
| LEAF_KEY=$(cat "$WORK_DIR/probe-leaf.key") | ||
| LEAF_CERT=$(cat "$WORK_DIR/probe-leaf.crt") | ||
| PROBE_CA=$(cat "$WORK_DIR/probe-ca.crt") | ||
|
|
||
| echo "Probe CA present in configured bundle -> chain must verify:" | ||
| printf '%s' "$PROBE_JS" | docker run -i --rm \ | ||
| -e PROBE_KEY="$LEAF_KEY" -e PROBE_LEAF="$LEAF_CERT" -e PROBE_CA="$PROBE_CA" \ | ||
| -e SSL_CERT_FILE=/tmp/probe-bundle.pem \ | ||
| --entrypoint /usr/local/bin/node "$IMAGE" - > "$WORK_DIR/accept.log" 2>&1 || true | ||
| cat "$WORK_DIR/accept.log" | ||
| if ! grep -q '^ACCEPTED' "$WORK_DIR/accept.log"; then | ||
| echo "::error::TLS verification does not follow the system CA store in $IMAGE" | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "Probe CA absent from configured bundle -> chain must be rejected:" | ||
| printf '%s' "$PROBE_JS" | docker run -i --rm \ | ||
| -e PROBE_KEY="$LEAF_KEY" -e PROBE_LEAF="$LEAF_CERT" \ | ||
| --entrypoint /usr/local/bin/node "$IMAGE" - > "$WORK_DIR/reject.log" 2>&1 || true | ||
| cat "$WORK_DIR/reject.log" | ||
| if ! grep -q '^REJECTED' "$WORK_DIR/reject.log"; then | ||
| echo "::error::untrusted probe CA was accepted via the default bundle in $IMAGE" | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "✅ CA trust verification passed for $IMAGE" | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
docker-build is the only gate that builds the real image from the Dockerfile, so the trust check lives here — it re-verifies every future base-image bump too, which is exactly how stale embedded trust data in a new base would get caught.