Repository navigation
Conversation
samsrabin
force-pushed
the
cirrus-runner-workflows
branch
from
August 6, 2025 21:21
00b95ee to
7114913
Compare
Member
Author
|
The ubuntu runner is currently failing because the |
samsrabin
force-pushed
the
cirrus-runner-workflows
branch
from
July 10, 2026 21:20
d4f93ae to
ee33e2c
Compare
Troubleshooting list-glade-cesm-input: Add `sleep 600`.
list-glade-cesm-input: Delete troubleshooting `sleep`.
…sldev-x86_64-almalinux9-gcc14-mpich:25.08.
It cannot: the image's PIO is built with PIO_ENABLE_FORTRAN=OFF, since it exists only to satisfy ESMF, and a case build needs the Fortran interface. The mpi-serial half is in the opposite position and is provable against the published image with no rebuild, because the wrappers prefer the mounted repo's macro file. The design doc had left this for the rebuild to discover. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A container case build compiles CTSM's mpi-serial submodule, so it links 2.5.4 while derecho links its 2.5.3 module and every version check calls that a match. MPI_SERIAL_PATH is the switch derecho uses for the same purpose. Repo-only on purpose: the published image already carries the prefix this names, and the wrappers prefer the mounted repo's copy of this file, so the mechanism can be proven against :20261007 before the serial-only rebuild commits to it. PIO is the other half and cannot be done this way -- the image's PIO is built without the Fortran interface. Not run; proving it needs a container case build on Casper, and the acceptance test is recorded at the site. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
derecho's parallelio-serial modulefile carries its full configure line, which settles the open question about timing and supplies the rest of the flags for 4b rather than leaving them to be guessed. Separately, CIME honors PIO_LIBDIR only when PIO_VERSION_MAJOR matches PIO_VERSION, and unexports it otherwise. derecho gets that variable from the module; the container has nothing to set it, so without an ENV the switch would fail silently and the submodule would be compiled while the macro looked right. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A container case build with only MPI_SERIAL_PATH set fails in PIO's configure with "Could NOT find MPISERIAL_C". The setting took effect as intended; what is unsafe is skipping CTSM's mpi-serial submodule build without also setting PIO_LIBDIR, which the published image cannot yet supply. The full mechanism is recorded at the site in gnu_container.cmake. The two settings now land together with the 4b rebuild. This also corrects NEXT_STEPS item 4b, which claimed MPI_SERIAL_PATH was the cheap half of that work, provable against the published image with no rebuild. Not compiled, built or run for this commit; the failing case build under cases_devcontainer is the evidence. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dockerfile.pio-fortran rebuilds ParallelIO with the Fortran interface on
top of :20261007 and exports the three variables CIME wants for an
installed PIO. It exists to test item 4b's switch before spending a
from-scratch rebuild on it; fold the layers in and delete the file
afterwards. It builds mpi-serial cases only.
gnu_container.cmake gains the consuming half, gated on $ENV{PIO} the way
derecho derives the same paths. The gate is what makes it safe to land
now: inert against the published image, armed against one that provides
a Fortran PIO.
Corrects three things this repo had wrong. What stops CIME building PIO
from source is PIO_VERSION_MAJOR in the environment, read by
buildlib.pio, not PIO_LIBDIR. PIO_TYPENAME_VALID_VALUES is mandatory
rather than optional polish, because buildlib.pio raises TypeError
without it. And the Dockerfile has no MPI_SERIAL_PATH assertion to
extend yet; the probe carries the one 4b should lift.
DRIFT_CHECK_DESIGN.md and README.md follow.
Not built or run: the probe needs a Casper allocation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
They name the paths item 4b's rebuilt image will have, so a container case build against :20261007 fails until that lands. Accepted deliberately: the alternative was gating the block on an environment variable the image sets, and a gate that can disarm itself silently reintroduces exactly the vacuous pass this item exists to end -- for the sake of a window that 4b closes. Dockerfile.pio-fortran has served its purpose; its header now records the result and that it is no longer re-runnable as it stands, since the macro points somewhere this image does not have. It goes away when 4b folds its layers in. Not built or run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Derecho's parallelio-serial is built with it on; the container will not be. The flag only adds library sources nothing in CTSM or CDEPS calls, and a static PIO never links them, so the executable item 5 compares is the same either way -- while turning it on would require netCDF-C dispatch support this image has not been checked for. Not built or run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Item 4b. The image carried an MPI stack (MPICH, parallel HDF5/netCDF/PnetCDF, two ESMF_COMM=mpich flavors) alongside a serial one. The MPI stack is gone, and the serial stack moves from /usr/local/serial to /usr/local, where ccs_config's container machine hardcodes NETCDF_PATH. PIO is now built with the Fortran interface, and the image exports PIO, PIO_VERSION_MAJOR and PIO_TYPENAME_VALID_VALUES, so a case build links the image's mpi-serial and PIO rather than compiling CTSM's libraries/mpi-serial and libraries/parallelio submodules. That is the drift this item exists to end: the two sides linked different mpi-serials while every version check reported a match. Deliberate divergences from derecho's parallelio-serial modulefile, both documented at the PIO step: BUILD_SHARED_LIBS and PIO_ENABLE_NETCDF_INTEGRATION stay off where derecho has them on. The wrapper scripts now inject --mpilib mpi-serial, because the container machine's default is mpich and the image no longer has one. Not built, not run. The mechanism was proven on Casper on 2026-10-09 with a throwaway probe image, folded in here and deleted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Captures the Casper build procedure for ctsm-ci-derecho-gnu: why a qsubbed build cannot also validate (podman's storage is node-local and dies with the job), the interactive and batch+load paths, the five validation wrappers, and the strings test on cesm.exe that is the real acceptance criterion. Also records the two failure modes a first rebuild after item 4b may hit, so neither gets debugged from scratch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cprnc failed at link on the first rebuild after item 4b, with ~200 undefined H5* symbols and a link line carrying only libnetcdff.a and libnetcdf.a. cime's cprnc build falls back to pkg_check_modules, which reads a .pc file's Libs: and not Libs.private, where netCDF's configure records the transitive -lhdf5_hl -lhdf5 -lz -lcurl -lxml2. That was harmless while /usr/local held the shared parallel stack, because DT_NEEDED supplied them; a static prefix has no equivalent, so the installed .pc files describe a link that cannot work. Rewriting the .pc files fixes every pkg-config consumer and leaves cime untouched. The layer asserts pkg-config --libs netcdf yields -lhdf5 afterwards, so a regression fails the image build rather than a case build. Verified only against representative .pc fixtures outside the image: the rewrite produces -lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lm -lz -lcurl -lxml2 in that order. Not yet rebuilt. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cprnc linked the static HDF5 correctly after the .pc fix and then died on undefined __truncsfhf2/__truncdfhf2/__truncxfhf2 -- libgcc _Float16 helpers that GCC 11.5.0 lacks. CMakeCache.txt recorded CMAKE_Fortran_COMPILER:FILEPATH=/usr/bin/f95. cprnc is built by cmake, the generated command line passes no -DCMAKE_*_COMPILER, and env_mach_specific.sh sets no compilers, so cmake used its built-in search for cc and f95. Neither name exists in /opt/gcc/<ver>/bin, which ships gcc and gfortran, and cmake searches by name before path. Item 4b removed ENV CC=mpicc/FC=mpifort along with the MPI stack. Those wrapped the same GCC the libraries are built with, so cprnc had been getting the right toolchain through them. The bare -lnetcdf failure masked the consequence until it was fixed. Absolute paths, asserted executable at build time, since a GCC_VERSION bump could otherwise leave them dangling with no earlier step noticing. smoke-test.sh checks the version each one reports. Not yet rebuilt. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
run_sys_tests puts compare and generate in required mutually exclusive groups, so a command carrying neither half of either pair dies in argparse before anything runs: run_sys_tests: error: one of the arguments -c/--compare --skip-compare is required The wrapper injected neither, and its own usage examples omitted them too, so every documented invocation failed that way. Skipping both is the right default here: a container validating a fresh image has no baselines to compare against, and /scratch dies with the allocation, so generating would discard the result. Passing -c or -g suppresses the matching injection, following the same ctsm_has_arg pattern as the other defaults. Verified with DRY_RUN: bare -t gets both skips; -c <name> gets only --skip-generate; --skip-compare -g <name> gets neither. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
create_newcase has no --overwrite and refuses an existing case directory, so the fixed --case brazil_test in the validation example failed on every run after the first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All four wrapper scripts and smoke-test.sh pass against the rebuilt image, and the acceptance test on cesm.exe is met: no /ctsm/libraries/mpi-serial or /ctsm/libraries/parallelio strings, and /usr/local/mpi-serial present, for both a test build and a standalone case build. Still unpublished; GHCR's newest tag predates the strip to one serial stack. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured on the item 4b build, 2026-10-15, replacing the two-stack image's 3.5 GB wherever that figure carried an argument. It does not change the case against building in GitHub Actions: the saving is 0.4 GB where the gap to a hosted runner's free disk is about an order of magnitude. The 3.9 GB archive and 54 GB podman-load peak are left as they were and labelled an upper bound; both were measured on the two-stack image and have not been remeasured. Also corrects the publish prerequisite from three validation scripts to smoke-test.sh plus the four wrappers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Bumps simple-build-create_test to ghcr.io/escomp/ctsm/ctsm-ci-derecho-gnu:20261015 and adds _Mmpi-serial to the test name it builds. Both in one commit because either alone leaves the job red. Since item 4b the image carries no MPICH, so a test name with no _M modifier takes the first <MPILIBS> entry of ccs_config's container machine, which is mpich. And :20261007 predates 4b, so it lacks the library paths gnu_container.cmake hardcodes. A commit carrying one half would fail CI for a reason unrelated to what it appears to change. simple-build-run_sys_tests is untouched: it runs in an unrelated mpich image, not this one. Item 4b is complete. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Revalidation, the cesm.exe acceptance test, the GHCR republish and the cirrus-testing.yml repin are all done and now read that way. The repinned CI job has not yet been exercised by an actual run, and the bullet says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
simple-build-run_sys_tests dies at git checkout with git's safe.directory protection, before it reaches CTSM; no workflow here sets safe.directory. The write-up notes the fix belongs inside the failing step rather than a step of its own, because this job runs under the k8s container hook where HOME can differ between steps. simple-build-create_test fails separately and its error is not yet captured; the item says so rather than guessing a shared cause. Neither is caused by item 4b, and the first is confirmed to predate the :20261015 repin. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The rebuild it was hedging against has happened: built, validated, published as :20261015 and pinned. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of changes
This PR is just to test GitHub Workflows on the CIRRUS cloud runners.