Add Tuolumne AMD (tamd): AFAR 24.3.0 amdflang + Cray MPICH, gfx942

[sbryngelson]

Adds tamd (LLNL Tuolumne, AFAR amdflang + Cray MPICH, OpenMP offload), the Tuolumne counterpart of famd. tuo (CCE) is unchanged.

Changes

• toolchain/modules: tamd uses the same modules as famd (python cmake, PrgEnv-amd, cpe/26.03), plus three additions that Tuolumne needs:
  • cray-pmi: the shared link line has -lpmi -lpmi2. Frontier loads this module by default; Tuolumne does not.
  • MFC_AMD_GPU_ARCH=gfx942: Tuolumne's GPUs are MI300A.
  • LIBRARY_PATH set to gcc-toolset-14: amdflang picks the newest gcc-toolset (15), which has no libquadmath.so on Tuolumne, so even a hello-world fails to link.
• toolchain/bootstrap/modules.sh: tamd shares the famd setup block (CRAY_MPICH_INC/LIB, hipfort, FC). Only the default OLCF_AFAR_ROOT differs. tamd is also added to the menu and to the CRAY_LD_LIBRARY_PATH exclusion.
• cmake/MFCTargets.cmake: the amdflang path hardcoded --offload-arch=gfx90a for compile, link, and the device stubs. It now reads MFC_AMD_GPU_ARCH and defaults to gfx90a, so famd and amdfund behave as before. With gfx90a on MI300A, omp_get_num_devices() returns 0.
• toolchain/templates/tamd.mako: tuo.mako with -c tamd, minus the MPICH_GPU_SUPPORT_ENABLED export (the shared block unsets it, as frontier_amd.mako does).

AFAR install

The default root is /p/lustre5/bryngelson1/afar/therock-afar-24.3.0-multiarch-10.1.0-592954c, readable by all users. Override it with OLCF_AFAR_ROOT.

The public 24.3.0 tarball has no flang-built MPI modules, and Cray's mpi.mod (classic flang) can't be read by flang 24. I built mpi, mpi_base, mpi_constants, mpi_sizeofs, and pmpi_base from MPICH 4.3.1 source with amdflang. They are installed at $OLCF_AFAR_ROOT/include/mpich4.3.1, the path famd reads. Cray MPICH 9.1.0 is based on MPICH 4.3.0a1; all 276 PARAMETER constants match Cray's mpif.h, which also has the Cray-only MPI_MODE_STREAM. MFC still links Cray's libmpifort_amd/libmpi_amd. Anyone installing another AFAR drop on Tuolumne needs to repeat this step.

Testing

On Tuolumne pdebug (1 node, 4× MI300A), built with ./mfc.sh build --gpu mp from a clean tree:

• syscheck, 4 ranks: MPI and OpenMP offload checks pass.
• benchmarks/5eq_rk3_weno3_hllc, 4 ranks / 4 GPUs: pre_process, simulation, and post_process all succeed (0.23 ns/gp/eq/rhs).
• ./mfc.sh test --gpu mp -% 10 --no-chemistry -- -c tamd: 71/71 passed, no retries.

Not tested: -e batch submission through tamd.mako, chemistry cases, and famd/amdfund after the arch change (the default keeps their flags identical).

---

Acknowledgement

• I confirm this PR meets the above expectations and reflects my own understanding and real-world context.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-11T03:47:42.511215Z	f01d526	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f01d526f08

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Register Flux before exposing the tamd batch template

On a Tuolumne login node exposing only Flux, ./mfc.sh run ... -e batch -c tamd cannot reach this script: toolchain/mfc/run/queues.py:84-95 checks only LSF, SLURM, and PBS and raises when none is active, despite this branch emitting Flux directives. Add a Flux queue implementation/detector (or otherwise submit this script through Flux); without it, the new batch path always aborts before queueing and only interactive allocations work.

AGENTS.md reference: AGENTS.md:L11-L11

Useful? React with 👍 / 👎.

[github-actions]

Claude Code Review

Head SHA: f01d526

Files changed:

• 4
• cmake/MFCTargets.cmake
• toolchain/bootstrap/modules.sh
• toolchain/modules
• toolchain/templates/tamd.mako

Findings:

• medium toolchain/bootstrap/modules.sh: The tamd default OLCF_AFAR_ROOT is /p/lustre5/bryngelson1/afar/..., a path in one person's home on Lustre. Other users on Tuolumne will get a missing or unreadable toolchain (PATH, FC, hipfort and MPICH include paths all derive from it) unless they override the variable. The famd default is a shared /sw/... path, and amdfund uses a similar personal path. Confidence: moderate, since I can't verify the permissions. Suggested fix: install AFAR to a shared project location, or fail early with a clear error if $OLCF_AFAR_ROOT/bin/amdflang doesn't exist.
• medium cmake/MFCTargets.cmake: MFC_AMD_GPU_ARCH is read from the environment only at CMake configure time. Changing it later, for example after switching between gfx90a and gfx942, does not re-trigger configure, so a stale --offload-arch stays in the cached build. This is low risk if builds start clean. Suggested fix: store it as a cache variable, or document that the build directory must be cleaned when the arch changes.
• medium toolchain/modules: tamd-all MFC_AMD_GPU_ARCH=gfx942 and the LIBRARY_PATH line are -all, so they also apply to CPU (-m c) loads. This is harmless, but the arch variable only matters for GPU builds. Low confidence that this is a problem.

(Checked: LIBRARY_PATH=...:$LIBRARY_PATH expands correctly through __export_assignments's eval. tamd.mako follows tuo.mako and drops MPICH_GPU_SUPPORT_ENABLED, which modules.sh unsets for tamd.)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.75%. Comparing base (a6add01) to head (f01d526).
⚠️ Report is 9 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2032   +/-   ##
=======================================
  Coverage   61.75%   61.75%           
=======================================
  Files          86       86           
  Lines       22787    22787           
  Branches     3357     3357           
=======================================
  Hits        14073    14073           
  Misses       6225     6225           
  Partials     2489     2489           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.