Skip to content

Fix flaky p0 ENOENT in test harness directory handling (#1968) - #1974

Draft
sbryngelson wants to merge 4 commits into
MFlowCode:masterfrom
sbryngelson:fix-test-p0-race
Draft

sbryngelson wants to merge 4 commits into
MFlowCode:masterfrom
sbryngelson:fix-test-p0-race

Conversation

@sbryngelson

@sbryngelson sbryngelson commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Description

Fixes #1968

The intermittent [Errno 2] No such file ... 'p0' came from directory handling in the test harness racing the filesystem. Evidence: full ./mfc.sh test --no-mpi --no-gpu -a -j 48 on Tuolumne (Lustre) without the fix gave 703 passed / 8 failed, and the failures included other filesystem-metadata errors ([Errno 116] Stale file handle on a D/ file, [Errno 39] Directory not empty from rmtree of D/). Test 903FA85A itself passed in that run, so I could not reproduce the exact p0 failure; the root cause is inferred from code and this error family, not captured directly.

Changes:

  • delete_directory (previous commit): rmtree that hits ENOENT mid-sweep for an entry already gone finishes the sweep and only fails if something remains. Unit test added; it fails on the old code.
  • test.py: the post-process silo scan used isdir then listdir on silo_hdf5/p0; replaced with a single glob so there is no window for the directory to vanish between the two calls.

Evidence

  • pytest toolchain/mfc/test_common.py: 3 passed with the fix; the new test fails (RED) with the old delete_directory.
  • Without fix, -a -j 48 full suite: 703 passed, 8 failed (Stale file handle, Directory not empty, 6x "Line count does not match" on QBMM / phase-change cases).
  • With fix, two consecutive ./mfc.sh test --no-mpi --no-gpu -a -j 48 full runs: 711 passed, 0 failed, 50 skipped each.
  • ./mfc.sh format and ./mfc.sh precheck pass.

Not verified: the original p0 failure was never observed here, so the green runs show the suite is stable with the change, not that this exact race is closed. The "Line count does not match" failures vanished in both fixed runs but I did not diagnose their cause (likely the same Lustre load, unconfirmed). GPU and MPI paths were not run.

Contribution Policy

All contributions are expected to demonstrate:

  • A clear understanding of the codebase
  • Alignment with product direction
  • Thoughtful reasoning behind changes
  • Evidence of real-world usage or hands-on experience with the problem

If these expectations are not met, we would prefer to implement the changes ourselves rather than spend time reviewing low-effort submissions.

@sbryngelson sbryngelson changed the title Tolerate ENOENT in test output cleanup (flaky 'No such file p0', #1968) Fix flaky p0 ENOENT in test harness directory handling (#1968) Oct 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Flaky test 903FA85A (1D Bubbles 2 Fluids): No such file 'p0' under -j 48

1 participant