Repository navigation
Fix flaky p0 ENOENT in test harness directory handling (#1968) - #1974
Draft
sbryngelson wants to merge 4 commits into
Draft
sbryngelson wants to merge 4 commits into
sbryngelson wants to merge 4 commits into
Conversation
sbryngelson
force-pushed
the
fix-test-p0-race
branch
from
October 10, 2026 16:27
51cc7cd to
0302e99
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #1968
The intermittent
[Errno 2] No such file ... 'p0'came from directory handling in the test harness racing the filesystem. Evidence: full./mfc.sh test --no-mpi --no-gpu -a -j 48on Tuolumne (Lustre) without the fix gave 703 passed / 8 failed, and the failures included other filesystem-metadata errors ([Errno 116] Stale file handleon a D/ file,[Errno 39] Directory not emptyfromrmtreeofD/). Test 903FA85A itself passed in that run, so I could not reproduce the exactp0failure; the root cause is inferred from code and this error family, not captured directly.Changes:
delete_directory(previous commit):rmtreethat hits ENOENT mid-sweep for an entry already gone finishes the sweep and only fails if something remains. Unit test added; it fails on the old code.test.py: the post-process silo scan usedisdirthenlistdironsilo_hdf5/p0; replaced with a singleglobso there is no window for the directory to vanish between the two calls.Evidence
pytest toolchain/mfc/test_common.py: 3 passed with the fix; the new test fails (RED) with the olddelete_directory.-a -j 48full suite: 703 passed, 8 failed (Stale file handle, Directory not empty, 6x "Line count does not match" on QBMM / phase-change cases)../mfc.sh test --no-mpi --no-gpu -a -j 48full runs: 711 passed, 0 failed, 50 skipped each../mfc.sh formatand./mfc.sh precheckpass.Not verified: the original
p0failure was never observed here, so the green runs show the suite is stable with the change, not that this exact race is closed. The "Line count does not match" failures vanished in both fixed runs but I did not diagnose their cause (likely the same Lustre load, unconfirmed). GPU and MPI paths were not run.Contribution Policy
All contributions are expected to demonstrate:
If these expectations are not met, we would prefer to implement the changes ourselves rather than spend time reviewing low-effort submissions.