Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -382,6 +382,9 @@ jobs:
GIT_HASH=${{ github.sha }}
SOCKET_SECURITY_MODE=${{ env.SOCKET_SECURITY_MODE }}

- name: Verify CA trust configuration

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docker-build is the only gate that builds the real image from the Dockerfile, so the trust check lives here — it re-verifies every future base-image bump too, which is exactly how stale embedded trust data in a new base would get caught.

run: ./scripts/verify-express-image-trust.sh bitgo/express:${{ github.sha }}

- name: Test Express Docker image
id: docker-test
run: |
Expand Down
18 changes: 17 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,19 @@ RUN \


FROM node:22.23.4-bookworm-slim@sha256:de170b39439eb0e5a3444ad7d60b015614a381e10fecae2e30ec9e735122f38d
RUN apt-get update && apt-get install -y tini
# The runtime validates TLS against the system CA bundle (NODE_OPTIONS at the
# bottom), so ca-certificates must be installed. Debian's package still trusts
# eMudhra roots after the October 2026 CA compromise, so they are stripped here
# until the distro removes them; scripts/verify-express-image-trust.sh asserts
# that none remain.
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates tini && rm -rf /var/lib/apt/lists/* && \
for f in $(find /usr/share/ca-certificates -name '*.crt'); do \
Comment on lines +39 to +44

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debian 12's ca-certificates (20250419~deb12u2) still ships eMudhra/emSign roots, so installing the package alone would swap one eMudhra trust path for another. This scrub is temporary and idempotent: once the distro drops the roots the loop and sed match nothing, and the verify script asserts the end state either way.

if openssl x509 -in "$f" -noout -subject 2>/dev/null | grep -qi 'emudhra\|emsign'; then \
echo "removing compromised trust anchor: $f" && rm "$f"; \
fi; \
done && \
sed -i -e '/emSign/d' -e '/emudhra/Id' /etc/ca-certificates.conf && \
update-ca-certificates --fresh
# copy the root node_modules to the bitgo-express parent node_modules
COPY --from=builder /tmp/bitgo/node_modules /var/node_modules/
COPY --from=builder /tmp/bitgo/modules/express /var/bitgo-express/
Expand Down Expand Up @@ -374,5 +386,9 @@ LABEL git_hash=${GIT_HASH}
USER node
ENV NODE_ENV=production
ENV BITGO_BIND=0.0.0.0
# Validate outbound TLS against the system CA bundle instead of Node's embedded
# root store, so OS trust updates (e.g. revoked roots) take effect.
ENV NODE_OPTIONS=--use-openssl-ca
Comment on lines +389 to +391

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NODE_EXTRA_CA_CERTS can only add anchors to the trusted set — it cannot remove compromised roots. So verification moves off Node's embedded store entirely: in Node's NewRootCertStore() --use-openssl-ca selects X509_STORE_set_default_paths as a mutually exclusive alternative to the bundled roots, and SSL_CERT_FILE pins those paths to the scrubbed bundle. That also means future OS trust updates take effect without a Node bump.

ENV SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
EXPOSE 3080
ENTRYPOINT ["/usr/bin/tini", "--", "/usr/local/bin/node", "/var/bitgo-express/bin/bitgo-express"]
12 changes: 12 additions & 0 deletions modules/express/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,18 @@ Environment: prod
Base URI: https://0.0.0.0:4000
```

### TLS trust

The BitGo Express image validates outbound TLS against the system CA
store (`/etc/ssl/certs/ca-certificates.crt`, from the Debian
`ca-certificates` package) instead of Node's embedded root store. The
container sets `NODE_OPTIONS=--use-openssl-ca` and
`SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt`. To trust an
additional CA (for example an internal CA), set `NODE_EXTRA_CA_CERTS`

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wording matters here and I verified it empirically against the built image: NODE_EXTRA_CA_CERTS extends the configured store (it does not replace it), while pointing SSL_CERT_FILE at another bundle replaces the default one.

to a PEM file that contains it — extra certificates are added on top
of the configured system store — or point `SSL_CERT_FILE` at a bundle
that includes it (this replaces the default bundle).

### Building the Docker container

To build the BitGo Express Docker container yourself from the source code, check out the latest **master** branch and run `docker build` from the project root.
Expand Down
114 changes: 114 additions & 0 deletions scripts/verify-express-image-trust.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
#!/bin/bash

# Verify a BitGo Express image cannot validate TLS chains rooted in eMudhra
# (October 2026 CA compromise): the system CA bundle carries no eMudhra/emSign
# trust, and TLS verification follows that bundle rather than Node's embedded
# root store.
#
# Usage: scripts/verify-express-image-trust.sh <image>

set -euo pipefail

IMAGE="${1:?usage: verify-express-image-trust.sh <image>}"

WORK_DIR=$(mktemp -d)
trap 'rm -rf "$WORK_DIR"' EXIT

echo "Verifying CA trust configuration of $IMAGE"

# The system CA bundle must exist and contain certificates.
if ! docker run --rm --entrypoint /bin/sh "$IMAGE" -c \
'test -s /etc/ssl/certs/ca-certificates.crt && grep -q "BEGIN CERTIFICATE" /etc/ssl/certs/ca-certificates.crt'; then
echo "::error::missing or empty /etc/ssl/certs/ca-certificates.crt in $IMAGE"
exit 1
fi

# The system bundle must carry no eMudhra/emSign trust. Node's embedded root
# store is reported but only matters when the runtime consults it, which the
# probes below assert it does not.
docker run --rm --entrypoint /usr/local/bin/node "$IMAGE" -e '
const fs = require("fs");
const tls = require("tls");
const { X509Certificate } = require("crypto");
const pem = fs.readFileSync("/etc/ssl/certs/ca-certificates.crt", "utf8");
const blocks = pem.match(/-----BEGIN CERTIFICATE-----[\s\S]*?-----END CERTIFICATE-----/g) || [];
const compromised = (x509) => /emudhra|emsign/i.test(`${x509.subject}\n${x509.issuer}`);
const embedded = tls.rootCertificates.map((block) => new X509Certificate(block));
console.log(`system bundle certs: ${blocks.length}, embedded roots: ${embedded.length}`);
const hits = blocks.map((block) => new X509Certificate(block)).filter(compromised);
if (hits.length > 0) {
console.error(`eMudhra/emSign trust found in the system bundle:\n${hits.map((x509) => x509.subject).join("\n")}`);
process.exit(1);
}
const embeddedHits = embedded.filter(compromised);
if (embeddedHits.length > 0) {
console.log(`note: Node embedded root store lists ${embeddedHits.length} eMudhra/emSign root(s); NODE_OPTIONS=--use-openssl-ca keeps them out of TLS verification (asserted by the probes below)`);
}
'

# TLS verification must follow the configured system bundle. A server whose

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The accept/reject pair is the real assertion: accept proves verification follows the configured bundle (dropping --use-openssl-ca breaks it, since the bundled roots then reject the probe CA), and reject proves the default bundle grants no trust by accident (removing the scrub or replacing the config with NODE_EXTRA_CA_CERTS alone breaks it). The embedded-store line is informational because that store is never consulted under the configured flags.

# chain is anchored in a throwaway probe CA is accepted only when that CA is in
# the bundle Node validates against; Node's embedded roots alone reject it.
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
-keyout "$WORK_DIR/probe-ca.key" -out "$WORK_DIR/probe-ca.crt" -days 1 \
-subj "/CN=Express Trust Probe CA"
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
-keyout "$WORK_DIR/probe-leaf.key" -out "$WORK_DIR/probe-leaf.csr" \
-subj "/CN=localhost" -addext "subjectAltName=DNS:localhost,IP:127.0.0.1"
openssl x509 -req -in "$WORK_DIR/probe-leaf.csr" \
-CA "$WORK_DIR/probe-ca.crt" -CAkey "$WORK_DIR/probe-ca.key" -CAcreateserial \
-out "$WORK_DIR/probe-leaf.crt" -days 1 -copy_extensions copy

PROBE_JS='
const fs = require("fs");
const tls = require("tls");
if (process.env.PROBE_CA) {
fs.writeFileSync("/tmp/probe-bundle.pem", process.env.PROBE_CA);
}
const server = tls.createServer(
{ key: process.env.PROBE_KEY, cert: process.env.PROBE_LEAF },
(socket) => socket.end("ok")
);
server.listen(0, "127.0.0.1", () => {
const socket = tls.connect(
{ host: "127.0.0.1", port: server.address().port, servername: "localhost", rejectUnauthorized: true },
() => {
console.log("ACCEPTED");
socket.end();
server.close();
}
);
socket.on("error", (err) => {
console.log(`REJECTED ${err.code || err.message}`);
server.close();
process.exitCode = 2;
});
});
'

LEAF_KEY=$(cat "$WORK_DIR/probe-leaf.key")
LEAF_CERT=$(cat "$WORK_DIR/probe-leaf.crt")
PROBE_CA=$(cat "$WORK_DIR/probe-ca.crt")

echo "Probe CA present in configured bundle -> chain must verify:"
printf '%s' "$PROBE_JS" | docker run -i --rm \
-e PROBE_KEY="$LEAF_KEY" -e PROBE_LEAF="$LEAF_CERT" -e PROBE_CA="$PROBE_CA" \
-e SSL_CERT_FILE=/tmp/probe-bundle.pem \
--entrypoint /usr/local/bin/node "$IMAGE" - > "$WORK_DIR/accept.log" 2>&1 || true
cat "$WORK_DIR/accept.log"
if ! grep -q '^ACCEPTED' "$WORK_DIR/accept.log"; then
echo "::error::TLS verification does not follow the system CA store in $IMAGE"
exit 1
fi

echo "Probe CA absent from configured bundle -> chain must be rejected:"
printf '%s' "$PROBE_JS" | docker run -i --rm \
-e PROBE_KEY="$LEAF_KEY" -e PROBE_LEAF="$LEAF_CERT" \
--entrypoint /usr/local/bin/node "$IMAGE" - > "$WORK_DIR/reject.log" 2>&1 || true
cat "$WORK_DIR/reject.log"
if ! grep -q '^REJECTED' "$WORK_DIR/reject.log"; then
echo "::error::untrusted probe CA was accepted via the default bundle in $IMAGE"
exit 1
fi

echo "✅ CA trust verification passed for $IMAGE"
Loading