Skip to content

fix: unblock hosted queues when an executor cannot start - #8

Merged
LBEILC merged 1 commit into
LBEILC:mainfrom
liuyinchu:fix/hosted-launch-failure
Oct 21, 2026
Merged

LBEILC merged 1 commit into
LBEILC:mainfrom
liuyinchu:fix/hosted-launch-failure

Conversation

@liuyinchu

Copy link
Copy Markdown
Contributor

Problem

A failed spawn leaves the first hosted job queued and retried on every scan, preventing healthy jobs behind it from starting. Reproduced with a real ENOENT spawn after the request's working directory disappears: repeated scans attempt [broken, broken, broken] rather than [broken, healthy], and waiters receive no terminal result.

Changes

  • Make launch failure terminal only while the app still owns the queued claim; atomically withdraw the claim before publishing failure.
  • Retain a recoverable failed marker and persist the exact terminal event before append. Recover interrupted partial UTF-8/newline appends without truncating bytes already observed by live followers or duplicating the result.
  • Isolate unknown corrupted logs so they remain unchanged and do not block startup or healthy jobs.
  • Preserve successful spawn ownership when parent log cleanup fails, and keep failed jobs identified as hosted in JobStore.
  • Document behavior in both READMEs.

This does not fix a successfully spawned executor that hangs/dies before claiming, arbitrary external log corruption, or unavailable filesystem writes.

Validation

  • 23 hosted tests pass (12 existing + 11 new), including real spawn, real CLI wait exit code 4, restart/partial-append recovery, and ownership cases.
  • Five targeted CLI environment/follower/claim/handoff tests pass. Independent review reproduced the recovery boundaries and found no remaining blocker.
  • Actual-launcher cleanup fault injection: baseline rejects after spawn; fixed preserves the successful PID.
  • Full app suite: baseline 150 passed / 19 failed / 3 skipped; patched 161 passed / 19 failed / 3 skipped. Exactly the same 19 missing-browser UI failures.
  • TypeScript, offline main-process esbuild bundling, release:check and diff checks pass. The complete desktop build remains unverified because its vendor-font network step could not be completed; offline bundling is not a complete build.
  • Full renderer suite retained baseline counts 234 passed / 67 failed / 123 skipped before the final app-only recovery correction; final app, hosted and CLI checks were rerun afterward.
  • No complete browser, Windows/macOS, or Electron UI validation is claimed.

The published tree matches the locally tested tree exactly: 11a7f258a31e07e2333991a320b260121a893e52. Separate from the checkpoint and existing config-lock changes.

AI disclosure

Implemented and independently reviewed with an OpenAI coding assistant. Reproductions and reported checks were actually run; environmental and validation limitations are explicit.

@LBEILC
LBEILC marked this pull request as ready for review October 21, 2026 06:56
@LBEILC
LBEILC merged commit 0a69b4b into LBEILC:main Oct 21, 2026
4 checks passed
@LBEILC

LBEILC commented Oct 21, 2026

Copy link
Copy Markdown
Owner

Merged, thank you! Turning the stuck first job into a terminal failure, without racing an executor that already owns the claim, is exactly the right boundary.

Locally on Windows 11 the hosted tests (including the 11 new ones) and the rest of the desktop suite pass, and CI passed on all three platforms. Two small notes for later, neither a blocker: .failed.json markers are never pruned (the same is true of the existing claimed/running markers), and a job whose events log is damaged will be reported on every 500 ms scan. If you feel like a follow-up, reporting that once per job would be welcome.

This will ship in 0.3.14, with credit in the release notes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants