Repository navigation
fix(launcher): A queued job is launched again when its runner never takes it - #53
Merged
Merged
Conversation
… it is replaced A runner launched for a queued job can idle out (15 minutes) before the job can run, for example while a job-level concurrency group is held by another job. Nothing checked the job again, so it stayed queued with no runner. Every launch, and every check that finds the job still queued and served, now sends the job's own message again, 300 s on. A check looks for a live instance or task tagged with the job id: if one serves it, the check goes on; if none does, the job is launched. The check ends when GitHub no longer has the job queued. Its one listing of the stack's machines also feeds the vCPU budget. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
… and cancelled jobs The fake GitHub's job status can now change within a test. The concurrency test ends with the group freeing: the job is taken and in progress, and its check schedules nothing. The started-or-finished guard also runs with a live runner tagged with the job. GitHub reports a cancelled job as status "completed" (probed on zeroae/ci-runner, run 43221273888, 2026-11-20). Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
The owner widened #51: a job whose runners never take it (the repo left the group, labels nothing matches) was relaunched about every 15 to 20 minutes for as long as GitHub kept it queued. A relaunch now waits at least 15 minutes after the job's last launch, the gap doubling for each launch after that, up to 60 minutes. A held concurrency group still frees into a runner within the hour, so the job is never given up. The message carries `launches` and `launched_at`; one without them (the webhook's, a swap's re-queue, a message from before this change) launches at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The backoff test replayed the last message whether or not a new one was sent, so a waiting check that dropped the job's message would have passed. Each check in it now must send exactly one next check, and the one too soon keeps the launch count and time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sodre
marked this pull request as ready for review
November 20, 2026 06:00
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #51
Summary
A queued job whose runner never gets it is no longer left waiting for ever. The launcher checks the job again, 300 seconds on, and launches a new runner when none is serving it. That covers a job held behind a job-level concurrency group, whose first runner idled out before the group freed. A job its runners never take is relaunched with a growing gap, at most once an hour.
The approach is a launcher re-check, not a sweeper poll. Every launch, and every check that finds the job still queued and served, sends the job's own message again, 300 seconds on. A check finds whether a live instance or task is tagged with the job id: if one is, the check goes on and launches nothing; if none is, the job is launched. The sweeper has no list of queued jobs or served repos, so it could not do this without new GitHub calls. The check ends when GitHub no longer reports the job queued. The one listing of the stack's machines feeds both the served check and the vCPU budget.
Relaunches back off (the widening): a job launched before, with nothing serving it, is relaunched no sooner than 15 minutes after its last launch, the gap doubling for each launch after that, up to 60 minutes. The message carries
launchesandlaunched_at; one without them (the webhook's, a swap's re-queue, a message from before this change) launches at once. A held concurrency group still frees into a runner within the hour, so no job is given up.Extras for the owner
Evidence
make test: 191 pass, 0 fail, 0 skippedsqswas[], where[checkLater(MSG)]was expectedcreateFleet: the check was never sentcreateFleetwhere one was expectedRunTaskran where the test expected none (expected: undefined)too soon at 5,2 !== 1; again with the gap forced to 0the check at 5 sends the next check,1 !== 2The red runs were on the launcher tests before the implementation: 8 failing, then 42 green in
test/launcher.test.js. Mutations after green: breaking the status check turns the drop tests red; breaking the served check turns the Fargate, concurrency and two-checks tests red.What else ran:
make test, 191 tests, 191 pass, 0 fail, 0 skipped (final run on 351e401).make build: 79238 of 1048576 bytes. A cancelled job's status was probed on zeroae/ci-runner, run 43221273888, 2026-11-20 (read only).Not run: the live check on the stack. It needs a deploy, and no criterion depends on it.
Merge Danger
Door: two-way. A revert restores the single launch-and-drop behaviour; nothing stored changes.
Blast Radius: the launcher, for every queued job. A job whose runners never take it is relaunched at most once an hour once the gap has grown, so it costs about one short-lived instance an hour while GitHub keeps it queued.
The merge changes the launcher Lambda only. It changes no image and no infrastructure, and needs a
make deploybefore it reaches the stack. No feature flag.Reviews
Correctness, Standards, Spec and Security, two rounds (the second a widening round): 11 findings, each with its outcome
Correctness, round 1
Standards, round 1
checkLaterreads like an action but only builds the parameters. Pushed back: its comment says what it builds, and a rename would cost a review round with no behaviour change.Spec, round 1
Spec's two "needs evidence" items were answered without a source change: the ticket's own table records the concurrency-held job's status as
queueduntil cancelled by hand on 2026-11-18, and docs/stack.md's launcher role line lists the queue's send.Security, the path list could not be read, so the session review ran: no findings.
Correctness, round 2
launched_atis not checked to be a number. Pushed back: only the launcher writes it, fromnow(); only a hand-edited message could carry another type.Standards, round 2: no findings.
Spec, round 2: no findings. The added criterion is MET, by the backoff test.
Security, round 2, the path list could not be read, so the session review ran: no findings.
Tracking
Epic: none
Spec: none
Ticket: #51 A job queued behind its concurrency group never gets a runner
Flag: none
🤖 Generated with Claude Code