Skip to content

feat(webhook)!: A job without a job label gets no runner, and swaps are no longer repaired - #58

Merged
sodre merged 6 commits into
mainfrom
t3/implement-issue-56
Nov 21, 2026
Merged

sodre merged 6 commits into
mainfrom
t3/implement-issue-56

Conversation

@sodre

@sodre sodre commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Closes #56

Summary

A job must now carry its job label, ci-job-<run_id>-<run_attempt>-<job key>-<matrix index>, to get a runner: a job of a ci-runner shape without one is ignored, and the webhook logs why. The webhook no longer repairs swaps. It logs a runner that runs a job other than its own, and needs no EC2 or ECS permission any more. If a swap still happens, the launcher's next check relaunches the job the runner was launched for.

webhook, on workflow_job queued
  fence 3, the shape (shapeOf)                          -> ignored: labels (no log, as before)
  jobLabelFault(labels, workflow_job)
    no ci-job- label                                    -> ignored: labels,    logged why "no job label"   (new)
    two, malformed, another run/attempt, empty, >255    -> ignored: job label, logged with its why        (as before)
  enqueue

webhook, on workflow_job in_progress
  runner named for this job, or not this stack's        -> ignored: no swap
  runner named for another job                          -> logged {"anomaly": "runner of another job", ...}, 200
     (before: re-queue its job from the machine's tags, re-tag the instance or task)

launcher, a check of a queued job (#51), widened
  every machine of the job: its runner busy on GitHub (GET /orgs/{org}/actions/runners/{runner-id})
    any idle (or no runner-id tag, or 404)              -> served, check again in 5 min (as before)
    all busy, job still queued on a second read         -> a swap: launch now, no gap, log `swapped`
    all busy, job no longer queued                      -> dropped
  the next check's message remembers `swapped`; a machine already in it waives the gap no more

Held for the owner: do not merge yet. The ticket's "Before this can start" isn't met. Every consumer runs-on naming a ci-runner capacity must carry the job label, the webhook's log must show no enqueued line without a ci-job- label for a week, and the owner must confirm. #55 merged on 2026-11-21 (UTC), so the week hasn't passed. On 2026-11-20 the owner chose to build this now and hold the merge. Until those hold, merging gives no runner to any job without the label.

What a reader should know:

  • Swaps end only once every job carries its label. This changes the ticket's framing. A runner's labels cover any job of its shape that has no job label. So a job that is ignored here can still take another job's runner while it waits in GitHub's queue. The docs, the glossary's "Swap" entry and ADR 0004 say so, which is why the precondition matters and why the anomaly line exists. Correctness review found this in round 1.
  • After a swap, the launcher relaunches the stranded job (widened). The machine's tags still name the job it was launched for. Until now the re-check counted that job as served until the machine ended. Now it asks GitHub whether the machine's runner is busy. A runner busy while its job is still queued runs another job, so the check launches the job a runner at once, as the old re-queue did. The owner widened Require the job label, and remove the swap repair #56 for this (below), though the ticket had put changes to A job queued behind its concurrency group never gets a runner #51's re-check out of scope.
  • Choices the ticket left open: a job without the label replies ignored: labels, as the ticket says, and is also logged with why: "no job label". Unlike a job outside our shape, it is ours, and its workflow waits forever. The in_progress reply for a swap is 200 logged: runner of another job. README's adoption examples now carry the label, since without it they would get no runner.

Extras for the owner

Evidence

Criterion Ticked Shown by
A queued job without a job label is ignored. A test covers it. yes a queued job without a job label is ignored, and the log says why
The swap repair and its tests are gone. A test shows an in_progress job on another job's runner is logged and not re-queued. yes a job in progress on a runner launched for another job is logged, and nothing is re-queued
The webhook role keeps no permission that only the swap repair used. yes the webhook touches no instance and no task, and knows no cluster
The docs, traps, glossary and ADR 0004 match the new behavior. yes Spec review, rounds 1 and 2, read docs/stack.md, docs/traps.md, docs/runbook.md, GLOSSARY.md, ADR 0004 and README.md and found them consistent with the code; the criteria check's leftover (a stale ECS_TAG_CHARS sentence in docs/stack.md) was fixed in f7de6f8
The existing test suite passes. yes node --test test/*.test.js
A check that finds every runner launched for the job busy while the job is still queued (a swap) launches the job a runner at once, without waiting out the relaunch gap. A test covers it. yes an instance / a task whose runner is busy while its job is still queued is a swap: the job is launched again at once
A check that finds the job's runner busy, but the job no longer queued on a second read, launches nothing. A test covers it. yes an instance / a task whose runner is busy because it just took its own job launches nothing
Test Red Green
a queued job without a job label is ignored, and the log says why before the change: [ 202, 'enqueued\n', 1 ] against [ 200, 'ignored: labels\n', 0 ]; on the final code, with the no-label line in jobLabelFault removed: TypeError ok
a job in progress on a runner launched for another job is logged, and nothing is re-queued against the swap repair: failed (it re-queued and re-tagged); with the launchedFor guard removed: failed (no anomaly line) ok
the webhook touches no instance and no task, and knows no cluster (template) against the old template: failed (five EC2/ECS statements and CLUSTER) ok
an instance / a task whose runner is busy while its job is still queued is a swap: the job is launched again at once (launcher) before the change: both failed (served, no launch); on the final code, with the busy check removed, and with the gap bypass removed: both failed ok
an instance / a task whose runner is busy because it just took its own job launches nothing before the change: both failed (a further check sent); with the second read removed: both failed ok
an instance / a task whose runner is idle still serves its job; a job is served while any one of its runners is idle, though another is busy passed at once on the old code, which served every live machine; they guard the new busy check against over-reach ok
a swap waives the relaunch gap once: a later check that finds only the same busy machine waits it out after round 3, before the fix: too soon: 20 of the 30 minutes after the second launch failed (launched) ok
a machine whose runner GitHub no longer has (404) still serves its job passed at once; with a 404 taken as busy: failed ok

node --test test/*.test.js on 3a7926f: 204 tests, 204 passed, 0 skipped, 0 failed, including the aws-emulator ECS tests. make build: build/template.yaml 76,706 of 1,048,576 bytes.

Removed with the repair:

  • the webhook's swap tests;
  • the ECS-emulator swap test;
  • the notifier's two "swapped runner is reported for the job it took" tests, which tested the re-tag;
  • the conventions tests for the DEFAULT_SHAPE, ECS_TAG_CHARS and ecsTag copies that only the repair used.

Not run: a live run on the stack (docs/runbook.md "Proving a change live"). It is covered with fakes, including whether the launcher's App token may read GET /orgs/{org}/actions/runners/{id}: it already mints and deletes runners through the same org API. The merge waits on the owner's preconditions, and once they hold the deploy itself is the live run.

Merge Danger

Door: two-way. A revert serves unlabelled jobs again and restores the swap repair along with its permissions.

Blast Radius: every queued job without a job label.

The merge changes the webhook and launcher Lambdas (inlined into the template) and the webhook role, which loses ec2:DescribeInstances, ec2:CreateTags, ecs:ListTasks, ecs:DescribeTasks and ecs:TagResource. The webhook also loses its CLUSTER variable. It deploys with make deploy. There is no image change and no flag. A consumer runs-on without the job label gets no runner from this deploy on, and waits "for a runner" forever. Deploy only once the ticket's preconditions hold.

Reviews

Correctness, Standards, Spec and Security, four rounds (the fourth at the owner's request): 16 findings, each with its outcome

Correctness, round 1 (f7de6f8, medium)

  1. A job without a job label is ignored but can still take a labelled runner of its shape, so "swaps can no longer happen" was wrong. Fixed: the comments, the glossary, stack.md, the traps, the runbook and ADR 0004 say swaps end only once every job carries its label.
  2. After a swap the launcher's re-check counts the swapped machine as serving the job it was launched for, so that job waits until the machine ends, and the comment claimed a relaunch. Fixed: the owner widened Require the job label, and remove the swap repair #56, and the check now relaunches a job whose runners are all busy while it is still queued.
  3. A reclaim of a swapped machine is no longer reported. Pushed back: the ticket removes the re-tag that made this possible. It needs a swap, which needs a job without the label. stack.md and the traps row say so.
  4. A launcher comment still said the webhook's re-tag carries an ECS_TAG_CHARS copy. Fixed.
  5. A launcher comment still said the webhook carries a tagsOf copy. Fixed.
  6. The traps row promised waiting, then launched, after a swap. Fixed: it says served until the swapped machine ends.
  7. A job without a job label replies ignored: labels, the same as one outside the shape. Pushed back: the ticket names ignored: labels, and the log line carries why: "no job label".

Standards, round 1

  1. The no-label check sat in the handler, duplicating the ignore-and-log block, and left jobLabelFault throwing on a job without a job label. Fixed: jobLabelFault reports no job label itself, and one block logs and replies.
  2. checkRunner named nothing it does. Fixed: renamed logSwap.
  3. The reply ignored: no swap names a concept the glossary said could not happen. Pushed back: a swap can still happen (finding 1), and the glossary defines it as what the webhook detects.

Spec, round 1: no findings.

Security, round 1 (run because this repo has no security-path list): no findings.

Correctness, round 2 (8ffa071, low): no findings. Standards, round 2: no findings. Spec, round 2: no findings. Security, round 2 (read origin/HEAD...8ffa071, the whole branch): no findings.

Correctness, round 3 (9836313, medium, widening: criteria 6 and 7)

  1. A machine that stays busy on another job made every later check skip the relaunch gap once the relaunch's runner idled out. Fixed: the next check's message remembers the swapped machines, and only a newly swapped one waives the gap (3a7926f, reviewed in round 4).
  2. GitHub's runner busy and the job's status may disagree for longer than the two reads, so a runner that just took its own job can read as a swap. Pushed back: the window is the moment the runner takes the job, a check lands there rarely, and the cost is one runner that idles out after 15 minutes.
  3. A non-404 error on the runner read fails the message. Pushed back: the job read already fails the message on any GitHub error, and the retry repeats the check.
  4. No test covered a runner GitHub answers 404. Fixed: a machine whose runner GitHub no longer has (404) still serves its job.
  5. No test covered a later check after a swap, or the swapped path under a full budget. Fixed for the later check: a swap waives the relaunch gap once. Pushed back for the budget: it only adds a log field.
  6. The runner reads run one after another. Pushed back: the loop stops at the first idle runner, and a job rarely has more than one serving machine.

Standards, round 3: no findings. Spec, round 3: no findings; both added criteria MET. Security, round 3: no findings.

Correctness, round 4 (3a7926f, low): no findings. Standards, round 4: no findings. Spec, round 4: no findings. Security, round 4: no findings.

Tracking

Epic: none
Spec: none
Ticket: #56 Require the job label, and remove the swap repair
Flag: none

🤖 Generated with Claude Code

sodre and others added 6 commits November 20, 2026 21:23
…ed, not repaired

A queued job in shape but without a ci-job- label is ignored (ignored:
labels) with a log line naming why. On in_progress, a runner of this
stack running another job than its own is logged as an anomaly; nothing
is re-queued or re-tagged. The webhook role loses ec2:DescribeInstances,
ec2:CreateTags, ecs:ListTasks, ecs:DescribeTasks and ecs:TagResource, and
the function loses CLUSTER.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
README's adoption examples carry the job label; the label contract,
stack.md, traps, the runbook's log lines, the glossary's Swap entry and
ADR 0004's rollout say a job without one gets no runner and a swap is
logged, not repaired.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…re-check does then

A runner's labels cover an unlabelled job of its shape, so swaps end only
once every job carries its label; the docs and comments said they could
no longer happen. After a swap the launcher logs `served` for the
stranded job until the swapped machine ends, not `waiting`/`launched`.
jobLabelFault reports the missing label itself, so it no longer throws
on a job without one; checkRunner is logSwap; stale launcher comments
about the webhook's tag copies are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er job

The webhook no longer re-tags a swapped machine, so its tags still name the
job it was launched for and the check counted that job as served until the
machine ended. Now a check asks GitHub whether each machine's runner is busy:
busy while the job is still queued, it runs another job. When every machine
of the job is so, the check reads the job again (its runner may have just
taken it) and launches it a runner at once, with no relaunch gap, logging
`swapped`. Widened into #56 by the owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A machine that stays busy on another job for hours made every later check
skip the back-off, once the relaunch's runner idled out. The next check's
message now remembers the swapped machines, and only a newly swapped one
waives the gap. Adds a test that a runner GitHub no longer has (404)
still counts as serving.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sodre
sodre marked this pull request as ready for review November 21, 2026 02:54
@sodre
sodre merged commit 8f8aa55 into main Nov 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Require the job label, and remove the swap repair

1 participant