Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions GLOSSARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,14 @@ The `runs-on` label that picks a CI runner's CPU architecture: `x64` or `arm64`.
A CI runner's capacity, size and architecture together. Runners of the same shape are interchangeable.

**Label contract**:
The labels a job must carry to be served: `self-hosted`, exactly one capacity label, exactly one size label and exactly one architecture label, plus `linux` if the job wants it. Nothing else is allowed. It holds on every capacity.
The labels a job must carry to be served: `self-hosted`, exactly one capacity label, exactly one size label and exactly one architecture label, plus `linux` if the job wants it, and a job label. Nothing else is allowed. It holds on every capacity. The job label is optional until the consumers carry it.

**Job label**:
The `runs-on` label unique to one job: `ci-job-<run_id>-<run_attempt>-<job key>-<matrix index>`. The CI runner launched for the job registers with it, so it can take no job with another job label. The job key is written by hand, since `github.job` is empty inside `runs-on`.
_Avoid_: run label (that names a run, shared by its jobs)

**Swap**:
A CI runner taking a queued job other than the one it was launched for. GitHub gives a job to any idle runner whose labels cover the job's, so swaps between runners of the same shape are normal.
A CI runner taking a queued job other than the one it was launched for. GitHub gives a job to any idle runner whose labels cover the job's, so swaps between runners of the same shape happen, but only of a job without a job label.

**Fargate capacity**:
Fargate Spot or Fargate On-Demand, the two capacities with no Docker daemon. A job that picks one promises not to use Docker.
Expand Down
2 changes: 2 additions & 0 deletions docs/adr/0003-tolerate-swaps-instead-of-run-labels.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Tolerate swaps instead of requiring a run label

Superseded by [ADR 0004](0004-a-job-label-pins-each-runner-to-its-job.md).

GitHub gives a queued job to any idle runner whose labels cover the job's, so a CI runner often runs a job other than the one it was launched for: a swap. We don't prevent swaps. The label contract requires a capacity, a size and an architecture label, so a swap only ever happens between runners of the same shape. The system repairs the bookkeeping instead. When the webhook gets `workflow_job` `in_progress`, it reads `runner_name`. If that runner was launched for another job, the webhook re-queues that other job, so it isn't stranded, and re-tags the runner's instance or task with the job it actually runs, so a Spot reclaim is reported on the right commit.

## Considered Options
Expand Down
22 changes: 22 additions & 0 deletions docs/adr/0004-a-job-label-pins-each-runner-to-its-job.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# A job label pins each runner to its job

Supersedes [ADR 0003](0003-tolerate-swaps-instead-of-run-labels.md).

GitHub gives a queued job to any idle runner whose labels cover the job's. ADR 0003 let that happen, so a CI runner often ran a job other than the one it was launched for (a swap), and the system repaired the bookkeeping afterwards. Swaps turned out to be routine (the webhook logged three in five seconds on 2026-11-20), and every piece built on top had to allow for them: the swap repair, and #51's re-check, which finds a job's machines by the job-id tag because the runner's name can't be trusted. We now prevent swaps instead.

Every job carries one label unique to it, the **job label**:

```
ci-job-<run_id>-<run_attempt>-<job key>-<matrix index>
```

The launcher registers each runner with its job's labels, so the runner launched for a job carries that job's label, and GitHub can give it no job with another job label. GitHub gives a job to a runner whose labels include all of the job's, so the runner can still take a job of its shape that has no job label: swaps end once every job carries one. The webhook accepts a job label only when it names the job's own run and attempt (`workflow_job.run_id` and `run_attempt`) and has something after them. A job with two, or with one for another run or attempt, is ignored.

The rest of the label is the consumer's to make unique within the run. GitHub can't do it for them: inside `runs-on`, `github.job` evaluates to an empty string (measured on 2026-11-20 in a scratch repo, by reading the labels of the queued jobs back from the API), so the job key is written by hand. `strategy.job-index` does work there, and is `0` in a job with no matrix. A reusable workflow sees only its own job keys and its own matrix, so two calls in one run would make the same label: the caller passes something unique into it, as an input.

Rollout is in two steps. First the label is accepted and checked but not required: a job without it is served as before, and swaps are still repaired, including a labelled job's runner taking a job without the label. Once the consumers carry it, the label is required and the swap repair is removed.

## Considered Options

- **Tolerate swaps and repair them** (ADR 0003). Reversed: the repair is code that only exists because runners are interchangeable, and it can't explain a runner that registers for a job and runs nothing anyone can find.
- **A run label only** (`ci-job-<run_id>-<run_attempt>`, as CodeBuild-hosted runners use). Rejected: jobs of one run would still swap with each other.
73 changes: 70 additions & 3 deletions docs/stack.md
Original file line number Diff line number Diff line change
Expand Up @@ -216,11 +216,77 @@ App's webhook points at), for each delivery:
`ci-spot`/`ci-ondemand`/`ci-fargate-spot`/`ci-fargate`, exactly one size label (`ci-cpu-2`, `-4`, `-8`,
`-16`), exactly one of `x64`/`arm64`, and optionally `linux`. None has a
default: a missing size or architecture label, an unknown `ci-cpu-*`, or
both `x64` and `arm64`, are outside it.
both `x64` and `arm64`, are outside it. A job label (below) is allowed on
top, and checked once the rest passes: a job with two, or one that leaves
out its run and attempt, names another run or attempt, has nothing after
them, or is over 255 characters, is ignored (`200`, `ignored: job label`)
and logged as `{"ignored": <repo>, "job_id", "label", "why"}`, `why` being
`two job labels`, `job label without its run and attempt`, `job label of
another run`, `job label of another attempt`, `job label names no job` or
`job label over 255 characters`.
7. Enqueues `{repo, job_id, run_id, head_sha, job_name, label, vcpus, arch,
labels, installation_id}` (`label` is the capacity label, lower case;
`vcpus` is 2, 4, 8 or 16; `arch` is `x64` or `arm64`) and answers `202`.

**The job label** (ADR-0004). A job may carry one label unique to it,
`ci-job-<run_id>-<run_attempt>-<job key>-<matrix index>`, compared
case-insensitively. The webhook accepts it only when it starts with
`ci-job-<run_id>-<run_attempt>-` of the event's `workflow_job` and has
something after that. It goes into the message's `labels` with the rest, the
launcher mints the runner's JIT config with those labels, and so the runner
launched for the job can take no job with another job label. GitHub gives a
job to a runner whose labels include all of the job's, so that runner can
still take a job of its shape that has no job label. It is optional for now:
a job without it is served as before, and swaps are still repaired (below).

A workflow writes it in `runs-on`, with the job key typed by hand: inside
`runs-on`, `github.job` evaluates to an empty string. `strategy.job-index` is
`0` in a job with no matrix. Measured on 2026-11-20 in a scratch repo, by
reading the queued jobs' labels back from the API:

```yaml
jobs:
tests:
runs-on: [self-hosted, ci-spot, ci-cpu-4, x64, "ci-job-${{ github.run_id }}-${{ github.run_attempt }}-tests-${{ strategy.job-index }}"]
```

A `runs-on` built by an expression (the adoption variable's fallback to a
GitHub-hosted runner) builds the label with `format`:

```yaml
runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON(format('["self-hosted","ci-spot","ci-cpu-4","x64","ci-job-{0}-{1}-tests-{2}"]', github.run_id, github.run_attempt, strategy.job-index)) || 'ubuntu-latest' }}
```

A reusable workflow (`workflow_call`) sees only its own job keys and its own
`strategy`, so a run that calls it twice, or from a matrix, gives both calls
the same label. Give it an input the caller makes unique, and put it in the
label:

```yaml
# the called workflow
on:
workflow_call:
inputs:
call: { type: string, required: true }
jobs:
e2e:
runs-on: ${{ fromJSON(format('["self-hosted","ci-spot","ci-cpu-4","x64","ci-job-{0}-{1}-e2e-{2}-{3}"]', github.run_id, github.run_attempt, strategy.job-index, inputs.call)) }}
# the caller
jobs:
e2e-a:
uses: ./.github/workflows/e2e.yml
with: { call: e2e-a }
```

A caller with a matrix passes `${{ strategy.job-index }}` in the input too.
GitHub refuses a runner label of 257 characters or more and silently cuts a
256-character one to 255, so the runner would not carry the job's label
(measured 2026-11-20 with `generate-jitconfig`); the webhook ignores a job
label over 255 characters. A 98-character job key is
accepted and a 100-character one is a workflow file error, so with an 11-digit
run id and a 3-digit matrix index a label stays under 130 characters, plus the
input.

**Telling the notifier a job succeeded.** On `workflow_job` `completed` with
conclusion `success`, and a `runner_name` that is `${stack}-<14-digit
stamp>-<job id>` (a job on a GitHub-hosted runner, another stack's, or none is
Expand All @@ -231,8 +297,9 @@ ignored, `200`), the webhook invokes the notifier asynchronously
(about 3 deliveries per job: `queued`, `in_progress`, `completed`), so nothing
about the App changes. The webhook does no GitHub call itself.

**Swap repair** (ADR-0003). GitHub gives a queued job to any idle runner whose
labels cover it, so a runner launched for job Y may take job X. Left alone,
**Swap repair** (ADR-0003, superseded by ADR-0004 once the job label is
required). GitHub gives a queued job to any idle runner whose labels cover it,
so a runner launched for job Y may take job X, unless X carries a job label. Left alone,
X's own message is later dropped as not queued, and Y never gets a runner.
On `workflow_job` `in_progress` the webhook reads `runner_name`:

Expand Down
4 changes: 2 additions & 2 deletions docs/traps.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@ Symptoms seen on the stack, and their causes. A quoted section name is in
| `permission denied` on `/var/run/docker.sock` | the host's `docker` group isn't gid 123 |
| `setup-python` fails, or picks up a broken Python | `AGENT_TOOLSDIRECTORY` isn't `/opt/hostedtoolcache` |
| A deploy waits 60 minutes, then fails on the state lock | an `apply` ran on Spot and was reclaimed; it must be `ci-ondemand` |
| A job sits "Waiting for a runner" | its capacity's vCPU budget is full, the repo isn't in the group, the App isn't installed on it, or the repo is public. Also check the label contract: `self-hosted`, exactly one of `ci-spot`/`ci-ondemand`/`ci-fargate-spot`/`ci-fargate`, exactly one size label (`ci-cpu-2`/`-4`/`-8`/`-16`), exactly one of `x64`/`arm64`, and otherwise only `linux`. The launcher's log tells the first case (`deferred`) apart from the rest (no line at all). A job held behind a concurrency group shows `served` while its runner is up; once that runner idles out, a later check (5 minutes apart, see "Launching for queued jobs" in `stack.md`) launches another, no sooner than 15 minutes after the last launch and backing off to an hour (`waiting` in the log). A job that queued before the repo was turned on stays waiting after the fix: cancel and re-run it |
| A job sits "Waiting for a runner" | its capacity's vCPU budget is full, the repo isn't in the group, the App isn't installed on it, or the repo is public. Also check the label contract: `self-hosted`, exactly one of `ci-spot`/`ci-ondemand`/`ci-fargate-spot`/`ci-fargate`, exactly one size label (`ci-cpu-2`/`-4`/`-8`/`-16`), exactly one of `x64`/`arm64`, and otherwise only `linux` and the job label. A job label (`ci-job-...`) must be the job's own run and attempt with a job key after them, and the only one: the webhook logs `ignored` with a `why` otherwise (see "The job label" in `stack.md`). The launcher's log tells the first case (`deferred`) apart from the rest (no line at all). A job held behind a concurrency group shows `served` while its runner is up; once that runner idles out, a later check (5 minutes apart, see "Launching for queued jobs" in `stack.md`) launches another, no sooner than 15 minutes after the last launch and backing off to an hour (`waiting` in the log). A job that queued before the repo was turned on stays waiting after the fix: cancel and re-run it |
| A job that ran on our runners before now sits "Waiting for a runner", and the webhook's log shows no `enqueued` line for it | missing size or architecture label: its `runs-on` names only its capacity (`self-hosted`, `ci-spot`). There are no defaults; add a size and an architecture label (`ci-cpu-4` and `x64` keep the old shape), then cancel and re-run the job |
| An `arm64` runner's instance comes up and terminates without taking its job, and its console shows `no matching manifest for linux/arm64` | `ghcr.io/zeroae/ci-runner:latest` has no arm64 entry yet: the image workflow hasn't pushed a multi-arch `:latest` since it began building arm64. Run the image workflow on `main`, then re-run the job |
| A job's runner was launched for another job (`$RUNNER_NAME` ends in another job id), and that other job sits "Waiting for a runner" | a swap. The webhook repairs it on the job's `in_progress` (its log says `swap`), and the launcher then launches a runner for the other job. No `swap` line: check the webhook's log for errors (its role needs `ec2:DescribeInstances` and `ec2:CreateTags`, and for a task `ecs:ListTasks`, `ecs:DescribeTasks` and `ecs:TagResource`); cancel and re-run the waiting job |
| A job's runner was launched for another job (`$RUNNER_NAME` ends in another job id), and that other job sits "Waiting for a runner" | a swap: the job the runner took has no job label (see "The job label" in `stack.md`). The webhook repairs it on the job's `in_progress` (its log says `swap`), and the launcher then launches a runner for the other job. No `swap` line: check the webhook's log for errors (its role needs `ec2:DescribeInstances` and `ec2:CreateTags`, and for a task `ecs:ListTasks`, `ecs:DescribeTasks` and `ecs:TagResource`); cancel and re-run the waiting job |
| A Fargate Spot job was lost (its task's `stopCode` is `SpotInterruption`) and its commit has no `ci-runner / <job>` check run | the notifier logged `ignored: <task id> is no job task of this stack` for every change of the stop: it found no job tags on the task. ECS keeps a stopped task's tags readable through `DescribeTasks` (measured live), so check the task's tags (`describe-tasks --include TAGS`, within an hour of the stop; `list-tags-for-resource` refuses a stopped task) and the notifier's role. Re-run the job |
| A job fails "The runner has received a shutdown signal", "lost communication" or "exit code 130", next to a failed `ci-runner / <job>` check | Spot reclaim: re-run the job |
| A job lost to a Spot reclaim stays in progress after its machine is gone; `gh run cancel` does nothing, and `gh run rerun` says `This workflow is already running` | GitHub didn't fail the job its runner died under. Let the run's other jobs finish (a force-cancel ends them all), then `gh api -X POST repos/zeroae/<repo>/actions/runs/<run id>/force-cancel` and re-run the failed jobs (see "Read a lost job") |
Expand Down
38 changes: 34 additions & 4 deletions functions/webhook.js
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
'use strict';
// webhook: GitHub workflow_job receiver behind a Function URL. Verifies the
// delivery, applies the label contract and fence 3, and enqueues the job for
// the launcher. It launches nothing itself, but repairs swaps (ADR-0003) and
// delivery, applies the label contract, the job label (ADR-0004) and fence 3,
// and enqueues the job for the launcher. It launches nothing itself, but
// repairs swaps of jobs without the job label (ADR-0003) and
// tells the notifier when a job of this stack's runners succeeds.
// Inlined into the template; keep it one dependency-free CommonJS file.
const crypto = require('node:crypto');
Expand Down Expand Up @@ -32,10 +33,11 @@ function signatureOk(key, raw, sig) {

// The label contract: `self-hosted`, exactly one capacity label, exactly one
// size label, exactly one architecture label, and optionally `linux`. None has
// a default, so a swap only happens between runners of the same shape.
// a default, so a swap only happens between runners of the same shape. A job
// label, checked by jobLabelFault after, is no part of the shape.
// Returns the job's shape, or null outside it.
function shapeOf(labels) {
const ls = (labels || []).map((l) => String(l).toLowerCase());
const ls = (labels || []).filter((l) => !isJobLabel(l)).map((l) => String(l).toLowerCase());
const of = (set) => ls.filter((l) => set.includes(l));
const capacities = of(CAPACITY);
const sizes = of(Object.keys(SIZE));
Expand All @@ -46,6 +48,29 @@ function shapeOf(labels) {
return { capacity: capacities[0], vcpus: SIZE[sizes[0]], arch: archs[0] };
}

// The job label (ADR-0004), `ci-job-<run_id>-<run_attempt>-<job key>-<matrix
// index>`: unique to one job, so the runner launched for the job, which
// registers with the job's labels, can take no other. Optional for now: a job
// without it is served as before. Returns why a job's labels break it, as
// { label, why }, or null.
const isJobLabel = (l) => String(l).toLowerCase().startsWith('ci-job-');
function jobLabelFault(labels, job) {
const ls = (labels || []).filter(isJobLabel).map(String);
if (!ls.length) return null;
if (ls.length > 1) return { label: ls.join(','), why: 'two job labels' };
const label = ls[0];
const m = label.toLowerCase().match(/^ci-job-(\d+)-(\d+)-(.*)$/);
if (!m) return { label, why: 'job label without its run and attempt' };
const [, run, attempt, rest] = m;
if (run !== String(job.run_id)) return { label, why: 'job label of another run' };
if (attempt !== String(job.run_attempt)) return { label, why: 'job label of another attempt' };
if (!rest) return { label, why: 'job label names no job' };
// GitHub refuses a longer runner label, and cuts a 256-character one to
// 255, so the runner would not carry it (generate-jitconfig, 2026-11-20).
if (label.length > 255) return { label, why: 'job label over 255 characters' };
return null;
}

// The job fields the instance or task carries, as tags.
const JOB_TAGS = { repo: 'repo', job_id: 'job-id', run_id: 'run-id', head_sha: 'head-sha', job_name: 'job-name' };
// ECS allows only these characters in a tag (EC2 allows any). A copy of the
Expand Down Expand Up @@ -160,6 +185,11 @@ function createHandler({ secret, queue, notifier, ec2, ecs, env, log = console.l
if (p.action !== 'queued') return reply(200, 'ignored: action');
const shape = shapeOf(job.labels);
if (!shape) return reply(200, 'ignored: labels');
const fault = jobLabelFault(job.labels, job);
if (fault) {
log(JSON.stringify({ ignored: repo.full_name, job_id: job.id, ...fault }));
return reply(200, 'ignored: job label');
}

// The shape as the queue message carries it. The message's `label` is the
// capacity label, as queued messages carry it.
Expand Down
8 changes: 8 additions & 0 deletions test/launcher.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,14 @@ test('a queued job mints a JIT config in the ci-runner group with its labels', a
assert.deepEqual(mint.body, { name: 'ci-runner-20261112120000-555', runner_group_id: 3, labels: ['self-hosted', 'ci-spot'], work_folder: '_work' });
});

test('the runner for a job with a job label registers with that label, so it can take no other job', async () => {
const labels = ['self-hosted', 'ci-spot', 'ci-cpu-4', 'x64', 'ci-job-444-2-tests-0'];
const { handler, gh } = setup();
await handler(sqsEvent({ ...MSG, labels }));
const mint = gh.calls.find((c) => c.key === 'POST /orgs/zeroae/actions/runners/generate-jitconfig');
assert.deepEqual(mint.body.labels, labels);
});

test('a queued job launches one Spot instance from the launch template, tagged with the job', async () => {
const { handler, ec2, logs } = setup();
assert.deepEqual(await handler(sqsEvent(MSG)), { batchItemFailures: [] });
Expand Down
Loading
Loading