Skip to content

tool-reap test: never hang the sidecar suite - #971

Closed
nedtwigg wants to merge 1 commit into
mainfrom
tool-reap-test-hang
Closed

nedtwigg wants to merge 1 commit into
mainfrom
tool-reap-test-hang

Conversation

@nedtwigg

@nedtwigg nedtwigg commented Oct 3, 2026

Copy link
Copy Markdown
Member

Main CI's Build & Test hung twice after #969 merged: about 70 min, then about 26 min on a re-run, before I cancelled each. The tree is identical to a PR head that passed. standalone/sidecar's node --test never finished, and tool-reap.test.js (added in #961) was the one file that never reported.

Mechanism:

  1. A real-PTY test that fails mid-run can leave its Tool process running.
  2. The Tool's setInterval keeps the PTY slave open.
  3. The node-pty handle keeps the file's process alive after every test has settled.
  4. node --test waits forever, so the failure itself is never printed.

Fix (test-only):

  • The test Tool exits on its own after 30 s.
  • After the last test, the file exits once results are reported.

A probe test that deliberately leaks a running Tool exits in 3 s with the backstop and hangs without it.

Not found: the CI-only failure underneath. The test passes 6/6 on macOS and in an Ubuntu 24.04 container (Node 24.18, zsh), alone and with the full sidecar suite under CPU load. With this change, the next occurrence prints the failing assertion instead of hanging.

Test plan

  • node --test tool-reap.test.js passes, 3 runs
  • Leak probe: the file exits in 3 s with the backstop, and hangs without it

🤖 Generated with Claude Code

Main CI's Build & Test hung twice after #969 merged (same tree as a passing PR head): standalone/sidecar's node --test never finished, and tool-reap.test.js was the one file that never reported. A real-PTY test that fails mid-run can leave its Tool running; the Tool's setInterval kept the PTY slave open, the PTY handle kept the file's process alive after every test settled, and node --test waits forever, so the failure itself is never printed.

- The test Tool exits on its own after 30 s instead of running forever.
- After the last test, the file exits once its results are reported.

A probe test that leaks a running Tool exits in 3 s with the backstop and hangs without it. The CI-only failure underneath is not reproduced (it passes 6/6 on macOS and in an Ubuntu 24.04 container, alone and under load); the next occurrence will now print it instead of hanging.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Deploying mouseterm with  Cloudflare Pages  Cloudflare Pages

Latest commit: 4914fdf
Status: ✅  Deploy successful!
Preview URL: https://89cb6425.mouseterm.pages.dev
Branch Preview URL: https://tool-reap-test-hang.mouseterm.pages.dev

View logs

@dormouse-bot dormouse-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This overlaps #970, which fixes the same main hang in the same file and edits the same setInterval line in TOOL, so the two conflict. Only one should land. #970 also reproduces the leak: 12 of 200 runs never exited after printing ok for every test, and the process holding the PTY was bash, alive in readline after a lost SIGHUP. The Tool had already exited. Against that mechanism, the 30 s setTimeout in the Tool has no effect; only the file-exit backstop would have unblocked those runs.

If this PR is the one kept, the test.after backstop is a hand-rolled node --test --test-force-exit, which #970 sets in standalone/sidecar/package.json. On Node 22, a file that fails a test and leaks an interval exits at once under that flag, still reporting the failure and exiting 1. The flag also covers every sidecar test file, not just this one.

@nedtwigg

nedtwigg commented Oct 3, 2026

Copy link
Copy Markdown
Member Author

Superseded by #970, which reproduces the hang (lost SIGHUP leaves bash holding the PTY), fixes the SIGINT-handler race this PR's CI surfaced, and bounds CI's Test step.

@nedtwigg nedtwigg closed this Oct 3, 2026
@nedtwigg
nedtwigg deleted the tool-reap-test-hang branch October 3, 2026 14:53

This branch is waiting to be deployed

1 waiting deployment
hosted-preview — 4914fdf3 Waiting Oct 3, 2026 by nedtwigg via cleanup #948
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants