Skip to content

test: triage never-run tests and run the survivors nightly - #437

Merged
arzafran merged 1 commit into
mainfrom
chore/test-triage
Oct 19, 2026
Merged

arzafran merged 1 commit into
mainfrom
chore/test-triage

Conversation

@arzafran

Copy link
Copy Markdown
Member

About 90 tests in the repo never ran in CI: 74 socket test files in tests_v2 and 17 UI test classes. This sorts them out. Tests that pass now run every night in a new workflow, obsolete ones are deleted, and failing ones stay in the tree with their reasons listed below so someone can look at them.

PR checks are unchanged. The runner script only behaves differently when the nightly workflow sets its new environment variables.

What changed

  • New .github/workflows/nightly-extended-tests.yml: runs every night and can be started by hand, with inputs to pick files, classes and the number of attempts. It builds the app once, runs tests_v2 in 3 shards, then the UI classes. A test that passes only on a retry is reported as FLAKY.
  • scripts/run-tests-v2-ci.sh gets optional settings for the test list, attempts, keep going after a failure, a per-test timeout and a results file.
  • Nightly lists: tests_v2/nightly_subset.txt (45 files) and scripts/nightly-ui-classes.txt (10 classes, 23 methods skipped).
  • 5 tests_v2 files fixed for current behaviour. Codex now installs PermissionRequest, and bare pane indexes are rejected.
  • UI fixes:
    • SidebarResizeUITests now launches tagged; untagged Debug launches are blocked, which made it hang for 19 minutes.
    • Its minimum-width assertion is now 200 to match the sidebar floor set in 29c2566. That is a contract change; please confirm the floor is intended.
    • WorkspaceDescriptionUITests no longer waits on a web-view setup step that the removed browser left behind.

Deleted (11 tests_v2 files)

  • test_signals_auto, test_terminal_input_render_report, test_ctrl_enter_keybind: obsolete.
  • test_cpu_usage, test_cpu_notifications: always printed "SKIP" and exited 0, so they never tested anything.
  • test_nested_split_does_not_disappear, test_new_tab_render_after_splits: they need more panes than the 4-pane cap allows, and other tests cover the same scenario.
  • Four test_command_palette_switcher_* files: they expect surface rows that only appear when searchAllSurfaces is on, which is off by default and which the tests never turn on.

Still failing, left in place and out of the nightly

These might be real bugs, so they were not deleted.

  • tests_v2:
    • test_focus_transition_workspace_split_race: a background split took the workspace selection, which breaks the socket focus policy.
    • test_workspace_split_title_updates: the new split isn't focused after surface.split.
    • test_review_diff_panel: counted 1 changed file where 2 were expected.
    • test_tmux_compat_geometry: list-panes reports pane_top -3.
    • test_notifications: 9 failures.
    • test_agent_screen_manifest_detection: the agent state never reached idle.
    • test_windows_api: surfaces report they aren't in a window on the CI runner.
    • Three command palette tests.
    • Two fuzz harnesses that hit the 300 s timeout.
    • test_codex_install_integration and test_new_tab_interactive_after_splits: the cause is unknown because their logs were cut off.
  • UI skips:
    • BonsplitTabDrag, 7 methods: minimal-mode layout assertions.
    • TerminalCmdClick, 10 methods: dash-prefixed and ls-style paths aren't resolved to the full path.
    • SidebarHelpMenu, 1 method.
    • CommandPaletteAllSurfaces, 4 methods: the app's socket never answered.
    • WorkspaceDescription, 1 method.
  • TitlebarShortcutHintsUITests: hint labels never show.

Test runs

That check ran a version of the workflow that started on push to this branch. The final file drops that trigger and has one more UI skip. GitHub only schedules a workflow once it is on main, so the first real nightly run happens after merge; start it by hand once to confirm.

@arzafran
arzafran merged commit 192fb64 into main Oct 19, 2026
12 checks passed
@arzafran
arzafran deleted the chore/test-triage branch October 19, 2026 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant