Skip to content

fix(launcher): A queued job is launched again when its runner never takes it - #53

Merged
sodre merged 4 commits into
mainfrom
t3/implement-issue-51
Nov 20, 2026
Merged

sodre merged 4 commits into
mainfrom
t3/implement-issue-51

Conversation

@sodre

@sodre sodre commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Closes #51

Summary

A queued job whose runner never gets it is no longer left waiting for ever. The launcher checks the job again, 300 seconds on, and launches a new runner when none is serving it. That covers a job held behind a job-level concurrency group, whose first runner idled out before the group freed. A job its runners never take is relaunched with a growing gap, at most once an hour.

The approach is a launcher re-check, not a sweeper poll. Every launch, and every check that finds the job still queued and served, sends the job's own message again, 300 seconds on. A check finds whether a live instance or task is tagged with the job id: if one is, the check goes on and launches nothing; if none is, the job is launched. The sweeper has no list of queued jobs or served repos, so it could not do this without new GitHub calls. The check ends when GitHub no longer reports the job queued. The one listing of the stack's machines feeds both the served check and the vCPU budget.

Relaunches back off (the widening): a job launched before, with nothing serving it, is relaunched no sooner than 15 minutes after its last launch, the gap doubling for each launch after that, up to 60 minutes. The message carries launches and launched_at; one without them (the webhook's, a swap's re-queue, a message from before this change) launches at once. A held concurrency group still frees into a runner within the hour, so no job is given up.

Extras for the owner

  • A queued job whose runner never takes it is relaunched about every 15 to 20 minutes. A launched runner idles for 900 seconds before it exits, so a check sees it as served until then. Relaunches continue for as long as GitHub keeps the job queued (up to its 24-hour window). No spec or source settles a cap or backoff.

Evidence

Criterion Ticked Shown by
A queued job with no runner after a launch gets a new launch yes test "a queued job whose launch produced no runner gets a new launch at its check"
A job held behind a concurrency group is launched once the group frees yes test "a job held behind a concurrency group keeps its runner while it is up, and is launched again once that runner has gone" (the fake's job status changes to in_progress when the group frees; the chain then ends)
A job already started or cancelled is not launched again yes test "a check for a job that has started, finished or was cancelled launches nothing and schedules no further check" (in_progress and completed; GitHub reports a cancelled job as completed)
Repeated checks do not launch two runners for one job yes test "two checks in a row launch one runner, not two"
Messages in the old shape are still handled yes test "a check of a message from before the size labels launches the default shape, and checks again"
The existing test suite passes yes make test: 191 pass, 0 fail, 0 skipped
A queued job whose runners never take it is relaunched with a growing gap, 15 minutes doubling to 60 yes test "a job whose runners never take it is relaunched with a growing gap: 15, then 30, then 60 minutes at most"
Test Red Green
Launch schedules its own check: the same message, five minutes on Before the change sqs was [], where [checkLater(MSG)] was expected ok
Launch produced no runner gets a new launch at its check Before the change no second createFleet: the check was never sent ok
Job held behind a concurrency group keeps its runner, then is launched again Before the change the check was never sent, so no relaunch ok
Started, finished or cancelled job launches nothing A guard: it passed on the code before this change, since the status check already dropped these. Broken on purpose (the status check removed), the drop tests went red ok
Two checks in a row launch one runner Before the change a third createFleet where one was expected ok
Old-shape message's check Before the change no check message was sent ok
Fargate job served by its own task is not launched again Before the change RunTask ran where the test expected none (expected: undefined) ok
A job its runners never take is relaunched with a growing gap Before the backoff: too soon at 5, 2 !== 1; again with the gap forced to 0 ok
A check too soon to relaunch still sends the next check With the waiting branch's send removed: the check at 5 sends the next check, 1 !== 2 ok

The red runs were on the launcher tests before the implementation: 8 failing, then 42 green in test/launcher.test.js. Mutations after green: breaking the status check turns the drop tests red; breaking the served check turns the Fargate, concurrency and two-checks tests red.

What else ran: make test, 191 tests, 191 pass, 0 fail, 0 skipped (final run on 351e401). make build: 79238 of 1048576 bytes. A cancelled job's status was probed on zeroae/ci-runner, run 43221273888, 2026-11-20 (read only).

Not run: the live check on the stack. It needs a deploy, and no criterion depends on it.

Merge Danger

Door: two-way. A revert restores the single launch-and-drop behaviour; nothing stored changes.

Blast Radius: the launcher, for every queued job. A job whose runners never take it is relaunched at most once an hour once the gap has grown, so it costs about one short-lived instance an hour while GitHub keeps it queued.

The merge changes the launcher Lambda only. It changes no image and no infrastructure, and needs a make deploy before it reaches the stack. No feature flag.

Reviews

Correctness, Standards, Spec and Security, two rounds (the second a widening round): 11 findings, each with its outcome

Correctness, round 1

  1. A queued job whose runner never takes it is relaunched about every 15 to 20 minutes for as long as GitHub keeps it queued. Fixed: the owner widened A job queued behind its concurrency group never gets a runner #51; relaunches now back off, no sooner than 15 minutes after the last launch, doubling to 60.
  2. Duplicate deliveries, retries after a failed send, and swap re-queues start another check chain for the same job. Pushed back: chains add checks and messages, never runners, because the served check guards every launch; docs/stack.md says there is no de-duplication beyond that check.
  3. Each check lists the stack's instances or tasks. Pushed back: the budget check lists the same machines for every message, so the served check adds no call.

Standards, round 1

  1. The message shape built by the check duplicates the budget deferral's send. Pushed back: the sends differ in delay (60 seconds against 300) and in purpose.
  2. The name checkLater reads like an action but only builds the parameters. Pushed back: its comment says what it builds, and a rename would cost a review round with no behaviour change.

Spec, round 1

  1. The two-checks test does not cover two deliveries arriving before the first runner is listed. Pushed back: criterion 4 is met as written; the window is the pre-existing one in finding 2.

Spec's two "needs evidence" items were answered without a source change: the ticket's own table records the concurrency-held job's status as queued until cancelled by hand on 2026-11-18, and docs/stack.md's launcher role line lists the queue's send.

Security, the path list could not be read, so the session review ran: no findings.

Correctness, round 2

  1. The backoff test replayed the last message whether or not a new one was sent, so a waiting check that dropped the job's message would pass. Fixed: every check in the test must send exactly one next check, and the one too soon keeps the launch count and time.
  2. Each check chain keeps its own backoff, so a job with two chains can relaunch more often than the gap says. Pushed back: a second chain comes only from a swap re-queue or a duplicate delivery, every chain is blocked while any runner serves the job, so launches stay a runner's lifetime apart; merging chains needs state outside the message, which the stack does not keep.
  3. A failed send after a launch leaves the retry with the old launch count. Pushed back: a rare SQS failure that costs one step of the backoff, still bounded by the served check.
  4. launched_at is not checked to be a number. Pushed back: only the launcher writes it, from now(); only a hand-edited message could carry another type.
  5. The relaunch time is computed twice, for the test and for the log line. Pushed back: one expression on adjacent lines, and the change would cost a review round.

Standards, round 2: no findings.

Spec, round 2: no findings. The added criterion is MET, by the backoff test.

Security, round 2, the path list could not be read, so the session review ran: no findings.

Tracking

Epic: none
Spec: none
Ticket: #51 A job queued behind its concurrency group never gets a runner
Flag: none

🤖 Generated with Claude Code

sodre and others added 4 commits November 20, 2026 00:36
… it is replaced

A runner launched for a queued job can idle out (15 minutes) before the job can
run, for example while a job-level concurrency group is held by another job.
Nothing checked the job again, so it stayed queued with no runner.

Every launch, and every check that finds the job still queued and served, now
sends the job's own message again, 300 s on. A check looks for a live instance
or task tagged with the job id: if one serves it, the check goes on; if none
does, the job is launched. The check ends when GitHub no longer has the job
queued. Its one listing of the stack's machines also feeds the vCPU budget.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
… and cancelled jobs

The fake GitHub's job status can now change within a test. The concurrency test
ends with the group freeing: the job is taken and in progress, and its check
schedules nothing. The started-or-finished guard also runs with a live runner
tagged with the job. GitHub reports a cancelled job as status "completed"
(probed on zeroae/ci-runner, run 43221273888, 2026-11-20).

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
The owner widened #51: a job whose runners never take it (the repo left the
group, labels nothing matches) was relaunched about every 15 to 20 minutes for
as long as GitHub kept it queued.

A relaunch now waits at least 15 minutes after the job's last launch, the gap
doubling for each launch after that, up to 60 minutes. A held concurrency group
still frees into a runner within the hour, so the job is never given up. The
message carries `launches` and `launched_at`; one without them (the webhook's,
a swap's re-queue, a message from before this change) launches at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The backoff test replayed the last message whether or not a new one was sent,
so a waiting check that dropped the job's message would have passed. Each check
in it now must send exactly one next check, and the one too soon keeps the
launch count and time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sodre
sodre marked this pull request as ready for review November 20, 2026 06:00
@sodre
sodre merged commit 6ca2f4e into main Nov 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A job queued behind its concurrency group never gets a runner

1 participant