Skip to content

jenkins-agent-terraform-2408: Node 24 LTS, eMudhra/emSign CA store refresh, Go 1.26.10 - #1175

Open
austindimmer wants to merge 2 commits into
opendevstack:masterfrom
austindimmer:fix/INFIAAS-11916-terraform-2408
Open

austindimmer wants to merge 2 commits into
opendevstack:masterfrom
austindimmer:fix/INFIAAS-11916-terraform-2408

Conversation

@austindimmer

Copy link
Copy Markdown

Refreshes every CA store in jenkins-agent-terraform-2408 after the eMudhra/emSign incident (eight compromised intermediate CAs, see https://emudhra.we-have-a-problem.com/ and GHSA-6q48-6c7w-9gw2), and moves Node off the EOL 20 line.

pip always loads the CA certificates from its vendored certifi in addition to the system trust store, so updating the OS ca-certificates alone is not enough.

Changes (common/jenkins-agents/terraform-2408/docker/Dockerfile.ubi9, python_requirements):

  • Node.js 20 -> 24 LTS (NodeSource setup_24.x, resolves 24.22.1)
  • uv 0.10.9 -> 0.13.5 (embedded CA store fix, GHSA-hq85-5cpj-mr62)
  • pip>=26.2.2 for every uv-managed Python, plus certifi>=2026.10.23
  • dnf -y update ca-certificates in the package layer
  • Go 1.21.13 -> 1.26.10; the tarball's SHA-256 (from go.dev) is now verified before extraction (ARG GO_SHA256)
  • Python 3.9 is no longer installed: it is EOL and no pip release with the fix supports it (pip >= 26.1 requires Python >= 3.10). It can still be installed on demand with uv python install 3.9.
  • Behaviour change: python3, python and python3-config are now the uv-managed default Python 3.12 (PYTHON_DEFAULT_VERSION, defined once), with the agent's libraries (boto3, requests, python-hcl2, certifi, ...). Before, python3 was the RHEL platform 3.9, which had those libraries only because uv pip install --system --python 3.9 resolved to it. Without this, scripts calling python3 would have lost them. /usr/local/bin now precedes /usr/bin on PATH. /usr/bin/python3 stays the platform Python for dnf/yum, which use it through their shebang; the build checks that dnf still works. python3-config is an exec wrapper, because the standalone build's script derives its prefix from its own path.
  • The console scripts (pipenv, virtualenv, az, pre-commit, ...) of the default Python are linked into /usr/local/bin, and the build checks them together with python3. pipenv now creates Python 3.12 environments by default (before: 3.9).
  • The Python install loop now fails the build on the first interpreter that fails. Before, fi; let a later successful iteration hide an earlier failure.
  • CHANGELOG: the python3/pipenv default under Changed, the CA refresh under Fixed, the Python 3.9 removal under Removed.
  • Docs: versions and python3 in inf-terraform.adoc; 2408 uses tenv, not tfenv.

Verified locally with the CI build command: node v24.22.1; pip 26.2.2 (vendored certifi 2026.10.19) and certifi 2026.10.23 on Python 3.11.17 and 3.12.15; python3 = 3.12.15 with the libraries; go1.26.10 (checksum OK); uv 0.13.5; pipenv, virtualenv, az and pre-commit on PATH; pipenv install / pipenv run smoke test green; dnf works. No CA store we own trusts the compromised intermediates, checked with the published emSign ECC Root CA - G3 cross-signatures as untrusted chain certificates (an older image of this agent fails the same check).

Known residuals (not changed here, follow-ups welcome): virtualenv==20.26.3 still seeds pip 24.1 into new virtualenvs (virtualenv >= 21.14.12 embeds pip 26.2.2); pipenv 2024.x, AWS SAM CLI (aws/aws-sam-cli#9359) and some Ruby gems (httpclient, excon, aws-sdk-core) ship their own older CA bundles.

Tasks:

  • Updated documentation in docs/modules/... directory
  • Ran tests in <quickstarter>/testdata directory (n/a: agent image only; CI job jenkins-agent-terraform-2408-ubi9 builds it)

Austin Dimmer added 2 commits October 26, 2026 14:55
…/emSign CA refresh, Go 1.26.10

The eMudhra/emSign incident (eight compromised intermediates) needs every CA
store in the agent refreshed: pip always loads its vendored certifi in
addition to the system store, so OS ca-certificates alone is not enough.

- NODEJS_VERSION 20 -> 24 (Node 20 is EOL; 24.22.1 ships the fixed root store)
- uv 0.10.9 -> 0.13.5 (embedded CA store fix)
- pip >= 26.2.2 and certifi >= 2026.10.23 for every uv-managed Python
- dnf update ca-certificates in the package layer
- GO_VERSION 1.21.13 -> 1.26.10, and the tarball's SHA-256 is verified
  (ARG GO_SHA256, from go.dev)
- drop Python 3.9: EOL, and no pip release with the fix supports it
  (pip >= 26.1 requires Python >= 3.10). uv is installed, so users can
  still `uv python install 3.9`
- the default Python (PYTHON_DEFAULT_VERSION=3.12, defined once) becomes
  python3, python and python3-config (an exec wrapper, because the script
  derives its prefix from its own path), and its console scripts (pipenv,
  virtualenv, az, pre-commit, ...) are linked into /usr/local/bin, which now
  precedes /usr/bin on PATH. Before, python3 and these scripts had the
  agent's libraries only because `--system --python 3.9` resolved to the
  RHEL platform python3.9. The build asserts python3, its libraries,
  python3-config, the CLIs, and that dnf still works on the platform python3
- the Python loop now fails the build on the first interpreter that fails;
  before, `fi;` let a later successful iteration mask an earlier failure
- CHANGELOG: the python3/pipenv default under Changed, the Python 3.9
  removal under Removed
- docs: Node/Python/Go versions, python3, tenv (not tfenv) for 2408
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant