Repository navigation
Make the screenshots and CI workflows ride out their flakes - #54
Merged
Merged
Conversation
On a runner, the screenshots test's tap on the search field sometimes left it without the keyboard, and typing the search then failed: "Neither element nor any descendant has keyboard focus", in two of the last five failed screenshots runs on main. Wait for the focus after the tap, and tap again if it doesn't come. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016kHgrqkADmcriUTgZBEMDP
The screenshots test saved failure.png and failure.txt only when it threw. When typing into the search field failed, XCTest recorded the failure without a throw, so there was no failure.png, and post_screenshots.sh took the run for a complete one: on 15 and 22 October, main's screenshots were replaced by a run's without the screens after the search. Save them in tearDown, whenever the test failed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016kHgrqkADmcriUTgZBEMDP
The screenshots workflow failed on main in four of its last ten runs,
each time in only one of the light and dark runs, which passed the next
day on the same code. Besides the search field without the keyboard,
the app once stopped answering XCTest ("App event loop idle notification
not received") for two minutes on the connection sheet, a tap on the
radio streams once opened nothing, and the simulator once failed to
launch the test runner. A second run of the failed appearance gets past
these, and its first run's result bundle is kept to see what went wrong.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016kHgrqkADmcriUTgZBEMDP
On 12 October, CI's app job booted its simulator, then xcodebuild said it couldn't find it, and listed no simulators at all, as though the simulator service had gone from under it. xcodebuild exits with 70 then, so try once more on that exit code; any other failure still fails the job straight away. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016kHgrqkADmcriUTgZBEMDP
ScreenshotsFrom run 39812249538, against a real Clementine. Left: main. Right: this pull request, light and dark.
Screens at a failure
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by John · project thread
The intermittent failures in the last month of runs, and what this does about each.
screenshots (failed on main in 4 of its last 10 runs, each time in only one of the light and dark runs):
failure.png, andpost_screenshots.shreplaced main's screenshots with a set missing the screens after the search. The failure files are now saved intearDown, for any failure.Screenshots-<appearance>-1.xcresult.ci (1 flaky failure in about 150 runs since 26 September):
Already fixed, and not seen again since:
givesUpWhenClementineIsGoneandDownloadTests.notOnTheLocalNetwork(fake Clementine fixes, 2 and 5 October), andConnectUITeststiming out while the simulator booted (fixed by booting it first). The String Catalog check's failures on 6 and 7 October were real: those commits were missing strings.Generated by Claude Code