Repository navigation
jenkins-agent-terraform-2306: Node 24 LTS, eMudhra/emSign CA store refresh - #1176
Open
austindimmer wants to merge 2 commits into
Open
austindimmer wants to merge 2 commits into
austindimmer wants to merge 2 commits into
Conversation
added 2 commits
October 26, 2026 14:55
…/emSign CA refresh The eMudhra/emSign incident (eight compromised intermediates) needs every CA store in the agent refreshed: pip always loads its vendored certifi in addition to the system store, so OS ca-certificates alone is not enough. - NODEJS_VERSION 18 -> 24 (Node 18 is EOL; 24.22.1 ships the fixed root store) - uv 0.10.9 -> 0.13.5 (embedded CA store fix) - pip >= 26.2.2 and certifi >= 2026.10.23 for the uv-managed Python - dnf update ca-certificates in the package layer - drop Python 3.8 and 3.9: EOL, and no pip release with the fix supports them (pip >= 26.1 requires Python >= 3.10); the 3.8 pip's vendored certifi 2024.02.02 still trusted the AAA cross-signed intermediates. uv is installed, so users can still `uv python install 3.8` / `3.9` - the default Python (PYTHON_DEFAULT_VERSION=3.11, defined once) becomes python3, python and python3-config (an exec wrapper, because the script derives its prefix from its own path), and its console scripts (pipenv, virtualenv, az, pre-commit, ...) are linked into /usr/local/bin, which already precedes /usr/bin on PATH. Before, python3 and these scripts had the agent's libraries only because `--system --python 3.9` resolved to the RHEL platform python3.9. The build asserts python3, its libraries, python3-config, the CLIs, and that dnf still works on the platform python3 - the Python loop now fails the build on the first interpreter that fails; before, `fi;` let a later successful iteration mask an earlier failure - CHANGELOG: the python3/pipenv default under Changed, the Python 3.8/3.9 removal under Removed - docs: Node/Python versions, python3 and uv-managed Pythons for 2306
1 of 2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refreshes every CA store in
jenkins-agent-terraform-2306after the eMudhra/emSign incident (eight compromised intermediate CAs, see https://emudhra.we-have-a-problem.com/ and GHSA-6q48-6c7w-9gw2), and moves Node off the EOL 18 line.pip always loads the CA certificates from its vendored certifi in addition to the system trust store, so updating the OS
ca-certificatesalone is not enough.Changes (
common/jenkins-agents/terraform-2306/docker/Dockerfile.ubi9,python_requirements):setup_24.x, resolves 24.22.1); nodemon and aws-cdk are unchangedpip>=26.2.2for the uv-managed Python, pluscertifi>=2026.10.23dnf -y update ca-certificatesin the package layeruv python install 3.8/3.9.python3,pythonandpython3-configare now the uv-managed default Python 3.11 (PYTHON_DEFAULT_VERSION, defined once), with the agent's libraries (boto3, requests, python-hcl2, certifi, ...). Before,python3was the RHEL platform 3.9, which had those libraries only becauseuv pip install --system --python 3.9resolved to it. Without this, scripts callingpython3would have lost them./usr/bin/python3stays the platform Python for dnf/yum, which use it through their shebang; the build checks that dnf still works.python3-configis an exec wrapper, because the standalone build's script derives its prefix from its own path./usr/local/bin, and the build checks them together withpython3.pipenvnow creates Python 3.11 environments where it used to pick 3.8.fi;let a later successful iteration hide an earlier failure.python3/pipenv default under Changed, the CA refresh under Fixed, the Python 3.8/3.9 removal under Removed.python3for 2306 ininf-terraform.adoc.Verified locally with the CI build command: node v24.22.1; pip 26.2.2 (vendored certifi 2026.10.19) and certifi 2026.10.23 on Python 3.11.17;
python3= 3.11.17 with the libraries; uv 0.13.5; pipenv, virtualenv, az and pre-commit on PATH;pipenv install/pipenv runworks; dnf works. No CA store we own trusts the compromised intermediates, checked with the published emSign ECC Root CA - G3 cross-signatures as untrusted chain certificates (an older image of the 2408 agent fails the same check).Known residuals (not changed here, follow-ups welcome):
virtualenv==20.26.3still seeds pip 24.1 into new virtualenvs (virtualenv >= 21.14.12 embeds pip 26.2.2); pipenv 2023.x, AWS SAM CLI (aws/aws-sam-cli#9359) and some Ruby gems (httpclient, excon, aws-sdk-core, google-api-client) ship their own older CA bundles.Tasks:
docs/modules/...directory<quickstarter>/testdatadirectory (n/a: agent image only; CI jobjenkins-agent-terraform-2306-ubi9builds it)