Skip to content

jenkins-agent-terraform-2306: Node 24 LTS, eMudhra/emSign CA store refresh - #1176

Open
austindimmer wants to merge 2 commits into
opendevstack:masterfrom
austindimmer:fix/INFIAAS-11916-terraform-2306
Open

austindimmer wants to merge 2 commits into
opendevstack:masterfrom
austindimmer:fix/INFIAAS-11916-terraform-2306

Conversation

@austindimmer

Copy link
Copy Markdown

Refreshes every CA store in jenkins-agent-terraform-2306 after the eMudhra/emSign incident (eight compromised intermediate CAs, see https://emudhra.we-have-a-problem.com/ and GHSA-6q48-6c7w-9gw2), and moves Node off the EOL 18 line.

pip always loads the CA certificates from its vendored certifi in addition to the system trust store, so updating the OS ca-certificates alone is not enough.

Changes (common/jenkins-agents/terraform-2306/docker/Dockerfile.ubi9, python_requirements):

  • Node.js 18 -> 24 LTS (NodeSource setup_24.x, resolves 24.22.1); nodemon and aws-cdk are unchanged
  • uv 0.10.9 -> 0.13.5 (embedded CA store fix, GHSA-hq85-5cpj-mr62)
  • pip>=26.2.2 for the uv-managed Python, plus certifi>=2026.10.23
  • dnf -y update ca-certificates in the package layer
  • Python 3.8 and 3.9 are no longer installed: they are EOL and no pip release with the fix supports them (pip >= 26.1 requires Python >= 3.10). The pip for 3.8 vendors certifi 2024.02.02, which still trusts the AAA cross-signed intermediates. Both can still be installed on demand with uv python install 3.8 / 3.9.
  • Behaviour change: python3, python and python3-config are now the uv-managed default Python 3.11 (PYTHON_DEFAULT_VERSION, defined once), with the agent's libraries (boto3, requests, python-hcl2, certifi, ...). Before, python3 was the RHEL platform 3.9, which had those libraries only because uv pip install --system --python 3.9 resolved to it. Without this, scripts calling python3 would have lost them. /usr/bin/python3 stays the platform Python for dnf/yum, which use it through their shebang; the build checks that dnf still works. python3-config is an exec wrapper, because the standalone build's script derives its prefix from its own path.
  • The console scripts of the default Python 3.11 (pipenv, virtualenv, az, pre-commit, pycodestyle, ...) are linked into /usr/local/bin, and the build checks them together with python3. pipenv now creates Python 3.11 environments where it used to pick 3.8.
  • The Python install loop now fails the build on the first interpreter that fails. Before, fi; let a later successful iteration hide an earlier failure.
  • CHANGELOG: the python3/pipenv default under Changed, the CA refresh under Fixed, the Python 3.8/3.9 removal under Removed.
  • Docs: Node/Python versions and python3 for 2306 in inf-terraform.adoc.

Verified locally with the CI build command: node v24.22.1; pip 26.2.2 (vendored certifi 2026.10.19) and certifi 2026.10.23 on Python 3.11.17; python3 = 3.11.17 with the libraries; uv 0.13.5; pipenv, virtualenv, az and pre-commit on PATH; pipenv install / pipenv run works; dnf works. No CA store we own trusts the compromised intermediates, checked with the published emSign ECC Root CA - G3 cross-signatures as untrusted chain certificates (an older image of the 2408 agent fails the same check).

Known residuals (not changed here, follow-ups welcome): virtualenv==20.26.3 still seeds pip 24.1 into new virtualenvs (virtualenv >= 21.14.12 embeds pip 26.2.2); pipenv 2023.x, AWS SAM CLI (aws/aws-sam-cli#9359) and some Ruby gems (httpclient, excon, aws-sdk-core, google-api-client) ship their own older CA bundles.

Tasks:

  • Updated documentation in docs/modules/... directory
  • Ran tests in <quickstarter>/testdata directory (n/a: agent image only; CI job jenkins-agent-terraform-2306-ubi9 builds it)

Austin Dimmer added 2 commits October 26, 2026 14:55
…/emSign CA refresh

The eMudhra/emSign incident (eight compromised intermediates) needs every CA
store in the agent refreshed: pip always loads its vendored certifi in
addition to the system store, so OS ca-certificates alone is not enough.

- NODEJS_VERSION 18 -> 24 (Node 18 is EOL; 24.22.1 ships the fixed root store)
- uv 0.10.9 -> 0.13.5 (embedded CA store fix)
- pip >= 26.2.2 and certifi >= 2026.10.23 for the uv-managed Python
- dnf update ca-certificates in the package layer
- drop Python 3.8 and 3.9: EOL, and no pip release with the fix supports
  them (pip >= 26.1 requires Python >= 3.10); the 3.8 pip's vendored
  certifi 2024.02.02 still trusted the AAA cross-signed intermediates.
  uv is installed, so users can still `uv python install 3.8` / `3.9`
- the default Python (PYTHON_DEFAULT_VERSION=3.11, defined once) becomes
  python3, python and python3-config (an exec wrapper, because the script
  derives its prefix from its own path), and its console scripts (pipenv,
  virtualenv, az, pre-commit, ...) are linked into /usr/local/bin, which
  already precedes /usr/bin on PATH. Before, python3 and these scripts had
  the agent's libraries only because `--system --python 3.9` resolved to the
  RHEL platform python3.9. The build asserts python3, its libraries,
  python3-config, the CLIs, and that dnf still works on the platform python3
- the Python loop now fails the build on the first interpreter that fails;
  before, `fi;` let a later successful iteration mask an earlier failure
- CHANGELOG: the python3/pipenv default under Changed, the Python 3.8/3.9
  removal under Removed
- docs: Node/Python versions, python3 and uv-managed Pythons for 2306
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant