Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions GLOSSARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,14 +26,14 @@ The `runs-on` label that picks a CI runner's CPU architecture: `x64` or `arm64`.
A CI runner's capacity, size and architecture together. Runners of the same shape are interchangeable.

**Label contract**:
The labels a job must carry to be served: `self-hosted`, exactly one capacity label, exactly one size label and exactly one architecture label, plus `linux` if the job wants it, and a job label. Nothing else is allowed. It holds on every capacity. The job label is optional until the consumers carry it.
The labels a job must carry to be served: `self-hosted`, exactly one capacity label, exactly one size label and exactly one architecture label, plus `linux` if the job wants it, and a job label. Nothing else is allowed. It holds on every capacity.

**Job label**:
The `runs-on` label unique to one job: `ci-job-<run_id>-<run_attempt>-<job key>-<matrix index>`. The CI runner launched for the job registers with it, so it can take no job with another job label. The job key is written by hand, since `github.job` is empty inside `runs-on`.
_Avoid_: run label (that names a run, shared by its jobs)

**Swap**:
A CI runner taking a queued job other than the one it was launched for. GitHub gives a job to any idle runner whose labels cover the job's, so swaps between runners of the same shape happen, but only of a job without a job label.
A CI runner taking a queued job other than the one it was launched for. GitHub gives a job only to a runner whose labels include all of the job's, and each runner carries its own job's job label, so no job that carries one can swap. A job of our shape without one gets no runner of its own but can still take another job's, so swaps end only once every such job carries its label. The webhook logs a swap as an anomaly, and repairs nothing.

**Fargate capacity**:
Fargate Spot or Fargate On-Demand, the two capacities with no Docker daemon. A job that picks one promises not to use Docker.
Expand Down
28 changes: 17 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,18 +62,24 @@ it's in both.
```yaml
jobs:
tests:
runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON('["self-hosted","ci-spot","ci-cpu-8","x64"]') || 'ubuntu-latest' }}
runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON(format('["self-hosted","ci-spot","ci-cpu-8","x64","ci-job-{0}-{1}-tests-{2}"]', github.run_id, github.run_attempt, strategy.job-index)) || 'ubuntu-latest' }}
build-arm:
runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON('["self-hosted","ci-spot","ci-cpu-4","arm64"]') || 'ubuntu-latest' }}
runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON(format('["self-hosted","ci-spot","ci-cpu-4","arm64","ci-job-{0}-{1}-build-arm-{2}"]', github.run_id, github.run_attempt, strategy.job-index)) || 'ubuntu-latest' }}
apply:
runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON('["self-hosted","ci-ondemand","ci-cpu-2","x64"]') || 'ubuntu-latest' }}
runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON(format('["self-hosted","ci-ondemand","ci-cpu-2","x64","ci-job-{0}-{1}-apply-{2}"]', github.run_id, github.run_attempt, strategy.job-index)) || 'ubuntu-latest' }}
lint:
runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON('["self-hosted","ci-fargate-spot","ci-cpu-2","arm64"]') || 'ubuntu-latest' }}
runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON(format('["self-hosted","ci-fargate-spot","ci-cpu-2","arm64","ci-job-{0}-{1}-lint-{2}"]', github.run_id, github.run_attempt, strategy.job-index)) || 'ubuntu-latest' }}
docs:
runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON('["self-hosted","ci-fargate","ci-cpu-4","x64"]') || 'ubuntu-latest' }}
runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON(format('["self-hosted","ci-fargate","ci-cpu-4","x64","ci-job-{0}-{1}-docs-{2}"]', github.run_id, github.run_attempt, strategy.job-index)) || 'ubuntu-latest' }}
```

Each `runs-on` names the full shape: capacity, size and architecture.
Each `runs-on` names the full shape: capacity, size and architecture, and
the job label, `ci-job-<run id>-<attempt>-<job key>-<matrix index>`, with
the job key typed by hand (`github.job` is empty inside `runs-on`). The
label pins the runner launched for the job to that job
([ADR-0004](docs/adr/0004-a-job-label-pins-each-runner-to-its-job.md));
a reusable workflow needs an input in it too (see "The job label" in
[docs/stack.md](docs/stack.md)).
A job on `ci-fargate-spot` or `ci-fargate` promises not to use Docker
(no `services:`, `container:`, Docker actions or `docker` commands): a
Fargate task has no Docker daemon
Expand All @@ -95,11 +101,11 @@ it's in both.

**The label contract:** `runs-on` must include `self-hosted`, exactly one of
`ci-spot`, `ci-ondemand`, `ci-fargate-spot` or `ci-fargate`, exactly one size label (`ci-cpu-2`, `ci-cpu-4`,
`ci-cpu-8`, `ci-cpu-16`) and exactly one of `x64` or `arm64`. It may also
include `linux`. None of them has a default. A job missing a size or an
architecture label, or with any other label (`aarch64`, an unknown
`ci-cpu-*`, two size labels, both `x64` and `arm64`), is ignored, and it waits
for a runner that never comes. Before changing the contract, `make consumers`
`ci-cpu-8`, `ci-cpu-16`), exactly one of `x64` or `arm64`, and exactly one
job label of its own run and attempt. It may also include `linux`. None of
them has a default. A job missing a size, an architecture or a job label, or
with any other label (`aarch64`, an unknown `ci-cpu-*`, two size labels, both
`x64` and `arm64`), is ignored, and it waits for a runner that never comes. Before changing the contract, `make consumers`
lists every job it affects.

**What the job gets:** Ubuntu 24.04, the runner as user `runner` (passwordless
Expand Down
6 changes: 3 additions & 3 deletions docs/adr/0004-a-job-label-pins-each-runner-to-its-job.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,17 @@ Supersedes [ADR 0003](0003-tolerate-swaps-instead-of-run-labels.md).

GitHub gives a queued job to any idle runner whose labels cover the job's. ADR 0003 let that happen, so a CI runner often ran a job other than the one it was launched for (a swap), and the system repaired the bookkeeping afterwards. Swaps turned out to be routine (the webhook logged three in five seconds on 2026-11-20), and every piece built on top had to allow for them: the swap repair, and #51's re-check, which finds a job's machines by the job-id tag because the runner's name can't be trusted. We now prevent swaps instead.

Every job carries one label unique to it, the **job label**:
Every job must carry one label unique to it, the **job label**:

```
ci-job-<run_id>-<run_attempt>-<job key>-<matrix index>
```

The launcher registers each runner with its job's labels, so the runner launched for a job carries that job's label, and GitHub can give it no job with another job label. GitHub gives a job to a runner whose labels include all of the job's, so the runner can still take a job of its shape that has no job label: swaps end once every job carries one. The webhook accepts a job label only when it names the job's own run and attempt (`workflow_job.run_id` and `run_attempt`) and has something after them. A job with two, or with one for another run or attempt, is ignored.
The launcher registers each runner with its job's labels, so the runner launched for a job carries that job's label, and GitHub can give it no job with another job label. GitHub gives a job to a runner whose labels include all of the job's, so a runner could still take a job of its shape that has no job label: that is why every job must carry one. The webhook accepts a job label only when it names the job's own run and attempt (`workflow_job.run_id` and `run_attempt`) and has something after them. A job with two, or with one for another run or attempt, is ignored.

The rest of the label is the consumer's to make unique within the run. GitHub can't do it for them: inside `runs-on`, `github.job` evaluates to an empty string (measured on 2026-11-20 in a scratch repo, by reading the labels of the queued jobs back from the API), so the job key is written by hand. `strategy.job-index` does work there, and is `0` in a job with no matrix. A reusable workflow sees only its own job keys and its own matrix, so two calls in one run would make the same label: the caller passes something unique into it, as an input.

Rollout is in two steps. First the label is accepted and checked but not required: a job without it is served as before, and swaps are still repaired, including a labelled job's runner taking a job without the label. Once the consumers carry it, the label is required and the swap repair is removed.
Rollout was in two steps. First the label was accepted and checked but not required: a job without it was served as before, and swaps were still repaired, including a labelled job's runner taking a job without the label. Once the consumers carried it, the label became required: a job of our shape without one is ignored, like any other job outside the label contract, and the webhook logs why. The swap repair is removed, and with it the webhook's EC2 and ECS permissions. A runner that runs a job other than its own, which only a job still without the label can make it do, is logged as an anomaly and not repaired; the job it was launched for is relaunched by #51's re-check, which takes a runner busy while its job is still queued as no longer serving it. The re-check also covers runners that never take their job for other reasons (a concurrency group, a failed boot).

## Considered Options

Expand Down
14 changes: 9 additions & 5 deletions docs/runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -352,17 +352,21 @@ $AWS logs tail /aws/lambda/ci-runner-notifier --since 1d

**The webhook** logs one JSON line per accepted job, `{"enqueued": "<repo>",
"job_id": ..., "label": ...}`, and `{"refused": "<repo>", "visibility": ...}`
for a job from any repo but a private `zeroae` one (fence 3), and `{"swap":
"<runner>", "instance": "i-...", "took": {...}, "requeued": {...}}` (with
`"task": "<task ARN>"` for a Fargate runner) for each swap it repairs: the runner launched for the `requeued` job started the `took`
job instead. Everything else it ignores without a log line.
for a job from any repo but a private `zeroae` one (fence 3), `{"ignored":
"<repo>", "job_id": ..., "why": ...}` for a job of our shape with no job label,
or a bad one (see "The job label" in `stack.md`), and `{"anomaly": "runner of
another job", "runner": ..., "launched_for": <job id>, "took": {...}}` if a
runner starts a job other than its own, which only a job without a job label
can make it do (`docs/traps.md`). Everything else it ignores without a log line.

**The launcher** logs one JSON line per job, carrying the job's fields plus one
of:

| Field | Meaning |
|---|---|
| `"launched": "i-..."`, `"runner"`, `"label"` | an instance is up for the job |
| `"launched": "i-..."`, `"runner"`, `"label"` | an instance is up for the job; with `"swapped": [...]`, because the machines launched for it ran other jobs (a swap, see `docs/traps.md`) |
| `"served": "i-..."` | a machine launched for the job is still up and its runner idle; the job is checked again in 5 minutes |
| `"waiting": "relaunch <n> from <time>"` | the job's runners never took it; it is relaunched no sooner than that |
| `"deferred": "20+8 of 24 spot vCPUs"` | the job's 8 vCPUs would take the capacity's 20 over its budget of 24; the job is back on the queue in 60 s |
| `"dropped": "completed"` (or another status) | the job no longer waits (cancelled, or taken by another runner); nothing launched |
| `"dropped": "job not found"` | GitHub answered 404 for the job |
Expand Down
Loading
Loading