Skip to content

Fail fast on Flux MPI launches outside an allocation - #1972

Draft
sbryngelson wants to merge 3 commits into
MFlowCode:masterfrom
sbryngelson:fix-flux-test-login
Draft

sbryngelson wants to merge 3 commits into
MFlowCode:masterfrom
sbryngelson:fix-flux-test-login

Conversation

@sbryngelson

@sbryngelson sbryngelson commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Description

Fixes #1967

On Flux systems, MPI launches outside an allocation (login node) submit a separate pbatch job per launch (default 12 h), so ./mfc.sh test queues one job per case and appears to hang. The templates now check FLUX_ENCLOSING_ID (set inside flux alloc/flux batch, unset on the login node) and stop with a clear message:

Not inside a Flux allocation: each launch would be submitted as its own batch job. Run inside 'flux alloc ...', use '-e batch', or pass --no-mpi.

  • toolchain/templates/include/helpers.mako: new require_flux_allocation helper.
  • tuo.mako: check for interactive MPI runs.
  • default.mako: same check when the selected launcher is srun and flux exists (on Tuolumne srun is the Flux wrapper; mfc.sh test uses the default template, which is what produced the pbatch jobs in the issue).

Testing

  • Login node, before (master): ./mfc.sh test -j 4 --no-gpu --only Cylinder launched pbatch jobs (named 1) via the flux srun wrapper (cancelled/failed immediately).
  • Login node, after: same command fails each case immediately with the message above; flux jobs shows no pending/running jobs.
  • Inside flux batch on pdebug: ./mfc.sh test -j 4 --no-gpu --only 6ED50013 passes.
  • Login node: ./mfc.sh test --no-mpi --no-gpu -j 4 --only 6ED50013 passes (unaffected).
  • ./mfc.sh format and ./mfc.sh precheck pass.

Not verified: mfc.sh run -c tuo interactive path directly (same helper), batch engine, non-Flux systems. With mfc.sh test each selected case reports the failure rather than aborting on the first (the harness's existing early-abort applies after 20 cases). Interactive run fails fast rather than only warning.

Contribution Policy

All contributions are expected to demonstrate:

  • A clear understanding of the codebase
  • Alignment with product direction
  • Thoughtful reasoning behind changes
  • Evidence of real-world usage or hands-on experience with the problem

If these expectations are not met, we would prefer to implement the changes ourselves rather than spend time reviewing low-effort submissions.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

mfc.sh test on a Tuolumne login node submits every test as a 12 h pbatch job

1 participant