Repository navigation
feat(webhook)!: A job without a job label gets no runner, and swaps are no longer repaired - #58
Merged
Merged
Conversation
…ed, not repaired A queued job in shape but without a ci-job- label is ignored (ignored: labels) with a log line naming why. On in_progress, a runner of this stack running another job than its own is logged as an anomaly; nothing is re-queued or re-tagged. The webhook role loses ec2:DescribeInstances, ec2:CreateTags, ecs:ListTasks, ecs:DescribeTasks and ecs:TagResource, and the function loses CLUSTER. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
README's adoption examples carry the job label; the label contract, stack.md, traps, the runbook's log lines, the glossary's Swap entry and ADR 0004's rollout say a job without one gets no runner and a swap is logged, not repaired. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…re-check does then A runner's labels cover an unlabelled job of its shape, so swaps end only once every job carries its label; the docs and comments said they could no longer happen. After a swap the launcher logs `served` for the stranded job until the swapped machine ends, not `waiting`/`launched`. jobLabelFault reports the missing label itself, so it no longer throws on a job without one; checkRunner is logSwap; stale launcher comments about the webhook's tag copies are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er job The webhook no longer re-tags a swapped machine, so its tags still name the job it was launched for and the check counted that job as served until the machine ended. Now a check asks GitHub whether each machine's runner is busy: busy while the job is still queued, it runs another job. When every machine of the job is so, the check reads the job again (its runner may have just taken it) and launches it a runner at once, with no relaunch gap, logging `swapped`. Widened into #56 by the owner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A machine that stays busy on another job for hours made every later check skip the back-off, once the relaunch's runner idled out. The next check's message now remembers the swapped machines, and only a newly swapped one waives the gap. Adds a test that a runner GitHub no longer has (404) still counts as serving. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sodre
marked this pull request as ready for review
November 21, 2026 02:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #56
Summary
A job must now carry its job label,
ci-job-<run_id>-<run_attempt>-<job key>-<matrix index>, to get a runner: a job of a ci-runner shape without one is ignored, and the webhook logs why. The webhook no longer repairs swaps. It logs a runner that runs a job other than its own, and needs no EC2 or ECS permission any more. If a swap still happens, the launcher's next check relaunches the job the runner was launched for.Held for the owner: do not merge yet. The ticket's "Before this can start" isn't met. Every consumer
runs-onnaming a ci-runner capacity must carry the job label, the webhook's log must show noenqueuedline without aci-job-label for a week, and the owner must confirm. #55 merged on 2026-11-21 (UTC), so the week hasn't passed. On 2026-11-20 the owner chose to build this now and hold the merge. Until those hold, merging gives no runner to any job without the label.What a reader should know:
ignored: labels, as the ticket says, and is also logged withwhy: "no job label". Unlike a job outside our shape, it is ours, and its workflow waits forever. Thein_progressreply for a swap is200 logged: runner of another job. README's adoption examples now carry the label, since without it they would get no runner.Extras for the owner
Evidence
in_progressjob on another job's runner is logged and not re-queued.docs/stack.md,docs/traps.md,docs/runbook.md,GLOSSARY.md, ADR 0004 andREADME.mdand found them consistent with the code; the criteria check's leftover (a staleECS_TAG_CHARSsentence indocs/stack.md) was fixed in f7de6f8node --test test/*.test.js[ 202, 'enqueued\n', 1 ]against[ 200, 'ignored: labels\n', 0 ]; on the final code, with the no-label line injobLabelFaultremoved:TypeErrorlaunchedForguard removed: failed (no anomaly line)CLUSTER)served, no launch); on the final code, with the busy check removed, and with the gap bypass removed: both failedtoo soon: 20 of the 30 minutes after the second launchfailed (launched)node --test test/*.test.json 3a7926f: 204 tests, 204 passed, 0 skipped, 0 failed, including the aws-emulator ECS tests.make build:build/template.yaml76,706 of 1,048,576 bytes.Removed with the repair:
DEFAULT_SHAPE,ECS_TAG_CHARSandecsTagcopies that only the repair used.Not run: a live run on the stack (
docs/runbook.md"Proving a change live"). It is covered with fakes, including whether the launcher's App token may readGET /orgs/{org}/actions/runners/{id}: it already mints and deletes runners through the same org API. The merge waits on the owner's preconditions, and once they hold the deploy itself is the live run.Merge Danger
Door: two-way. A revert serves unlabelled jobs again and restores the swap repair along with its permissions.
Blast Radius: every queued job without a job label.
The merge changes the webhook and launcher Lambdas (inlined into the template) and the webhook role, which loses
ec2:DescribeInstances,ec2:CreateTags,ecs:ListTasks,ecs:DescribeTasksandecs:TagResource. The webhook also loses itsCLUSTERvariable. It deploys withmake deploy. There is no image change and no flag. A consumerruns-onwithout the job label gets no runner from this deploy on, and waits "for a runner" forever. Deploy only once the ticket's preconditions hold.Reviews
Correctness, Standards, Spec and Security, four rounds (the fourth at the owner's request): 16 findings, each with its outcome
Correctness, round 1 (f7de6f8, medium)
stack.md, the traps, the runbook and ADR 0004 say swaps end only once every job carries its label.stack.mdand the traps row say so.ECS_TAG_CHARScopy. Fixed.tagsOfcopy. Fixed.waiting, thenlaunched, after a swap. Fixed: it saysserveduntil the swapped machine ends.ignored: labels, the same as one outside the shape. Pushed back: the ticket namesignored: labels, and the log line carrieswhy: "no job label".Standards, round 1
jobLabelFaultthrowing on a job without a job label. Fixed:jobLabelFaultreportsno job labelitself, and one block logs and replies.checkRunnernamed nothing it does. Fixed: renamedlogSwap.ignored: no swapnames a concept the glossary said could not happen. Pushed back: a swap can still happen (finding 1), and the glossary defines it as what the webhook detects.Spec, round 1: no findings.
Security, round 1 (run because this repo has no security-path list): no findings.
Correctness, round 2 (8ffa071, low): no findings. Standards, round 2: no findings. Spec, round 2: no findings. Security, round 2 (read
origin/HEAD...8ffa071, the whole branch): no findings.Correctness, round 3 (9836313, medium, widening: criteria 6 and 7)
busyand the job'sstatusmay disagree for longer than the two reads, so a runner that just took its own job can read as a swap. Pushed back: the window is the moment the runner takes the job, a check lands there rarely, and the cost is one runner that idles out after 15 minutes.Standards, round 3: no findings. Spec, round 3: no findings; both added criteria MET. Security, round 3: no findings.
Correctness, round 4 (3a7926f, low): no findings. Standards, round 4: no findings. Spec, round 4: no findings. Security, round 4: no findings.
Tracking
Epic: none
Spec: none
Ticket: #56 Require the job label, and remove the swap repair
Flag: none
🤖 Generated with Claude Code