diff --git a/GLOSSARY.md b/GLOSSARY.md index bc87760..09fc278 100644 --- a/GLOSSARY.md +++ b/GLOSSARY.md @@ -26,14 +26,14 @@ The `runs-on` label that picks a CI runner's CPU architecture: `x64` or `arm64`. A CI runner's capacity, size and architecture together. Runners of the same shape are interchangeable. **Label contract**: -The labels a job must carry to be served: `self-hosted`, exactly one capacity label, exactly one size label and exactly one architecture label, plus `linux` if the job wants it, and a job label. Nothing else is allowed. It holds on every capacity. The job label is optional until the consumers carry it. +The labels a job must carry to be served: `self-hosted`, exactly one capacity label, exactly one size label and exactly one architecture label, plus `linux` if the job wants it, and a job label. Nothing else is allowed. It holds on every capacity. **Job label**: The `runs-on` label unique to one job: `ci-job----`. The CI runner launched for the job registers with it, so it can take no job with another job label. The job key is written by hand, since `github.job` is empty inside `runs-on`. _Avoid_: run label (that names a run, shared by its jobs) **Swap**: -A CI runner taking a queued job other than the one it was launched for. GitHub gives a job to any idle runner whose labels cover the job's, so swaps between runners of the same shape happen, but only of a job without a job label. +A CI runner taking a queued job other than the one it was launched for. GitHub gives a job only to a runner whose labels include all of the job's, and each runner carries its own job's job label, so no job that carries one can swap. A job of our shape without one gets no runner of its own but can still take another job's, so swaps end only once every such job carries its label. The webhook logs a swap as an anomaly, and repairs nothing. **Fargate capacity**: Fargate Spot or Fargate On-Demand, the two capacities with no Docker daemon. A job that picks one promises not to use Docker. diff --git a/README.md b/README.md index 0f115a6..21b528c 100644 --- a/README.md +++ b/README.md @@ -62,18 +62,24 @@ it's in both. ```yaml jobs: tests: - runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON('["self-hosted","ci-spot","ci-cpu-8","x64"]') || 'ubuntu-latest' }} + runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON(format('["self-hosted","ci-spot","ci-cpu-8","x64","ci-job-{0}-{1}-tests-{2}"]', github.run_id, github.run_attempt, strategy.job-index)) || 'ubuntu-latest' }} build-arm: - runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON('["self-hosted","ci-spot","ci-cpu-4","arm64"]') || 'ubuntu-latest' }} + runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON(format('["self-hosted","ci-spot","ci-cpu-4","arm64","ci-job-{0}-{1}-build-arm-{2}"]', github.run_id, github.run_attempt, strategy.job-index)) || 'ubuntu-latest' }} apply: - runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON('["self-hosted","ci-ondemand","ci-cpu-2","x64"]') || 'ubuntu-latest' }} + runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON(format('["self-hosted","ci-ondemand","ci-cpu-2","x64","ci-job-{0}-{1}-apply-{2}"]', github.run_id, github.run_attempt, strategy.job-index)) || 'ubuntu-latest' }} lint: - runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON('["self-hosted","ci-fargate-spot","ci-cpu-2","arm64"]') || 'ubuntu-latest' }} + runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON(format('["self-hosted","ci-fargate-spot","ci-cpu-2","arm64","ci-job-{0}-{1}-lint-{2}"]', github.run_id, github.run_attempt, strategy.job-index)) || 'ubuntu-latest' }} docs: - runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON('["self-hosted","ci-fargate","ci-cpu-4","x64"]') || 'ubuntu-latest' }} + runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON(format('["self-hosted","ci-fargate","ci-cpu-4","x64","ci-job-{0}-{1}-docs-{2}"]', github.run_id, github.run_attempt, strategy.job-index)) || 'ubuntu-latest' }} ``` - Each `runs-on` names the full shape: capacity, size and architecture. + Each `runs-on` names the full shape: capacity, size and architecture, and + the job label, `ci-job----`, with + the job key typed by hand (`github.job` is empty inside `runs-on`). The + label pins the runner launched for the job to that job + ([ADR-0004](docs/adr/0004-a-job-label-pins-each-runner-to-its-job.md)); + a reusable workflow needs an input in it too (see "The job label" in + [docs/stack.md](docs/stack.md)). A job on `ci-fargate-spot` or `ci-fargate` promises not to use Docker (no `services:`, `container:`, Docker actions or `docker` commands): a Fargate task has no Docker daemon @@ -95,11 +101,11 @@ it's in both. **The label contract:** `runs-on` must include `self-hosted`, exactly one of `ci-spot`, `ci-ondemand`, `ci-fargate-spot` or `ci-fargate`, exactly one size label (`ci-cpu-2`, `ci-cpu-4`, -`ci-cpu-8`, `ci-cpu-16`) and exactly one of `x64` or `arm64`. It may also -include `linux`. None of them has a default. A job missing a size or an -architecture label, or with any other label (`aarch64`, an unknown -`ci-cpu-*`, two size labels, both `x64` and `arm64`), is ignored, and it waits -for a runner that never comes. Before changing the contract, `make consumers` +`ci-cpu-8`, `ci-cpu-16`), exactly one of `x64` or `arm64`, and exactly one +job label of its own run and attempt. It may also include `linux`. None of +them has a default. A job missing a size, an architecture or a job label, or +with any other label (`aarch64`, an unknown `ci-cpu-*`, two size labels, both +`x64` and `arm64`), is ignored, and it waits for a runner that never comes. Before changing the contract, `make consumers` lists every job it affects. **What the job gets:** Ubuntu 24.04, the runner as user `runner` (passwordless diff --git a/docs/adr/0004-a-job-label-pins-each-runner-to-its-job.md b/docs/adr/0004-a-job-label-pins-each-runner-to-its-job.md index fc8a611..11c3297 100644 --- a/docs/adr/0004-a-job-label-pins-each-runner-to-its-job.md +++ b/docs/adr/0004-a-job-label-pins-each-runner-to-its-job.md @@ -4,17 +4,17 @@ Supersedes [ADR 0003](0003-tolerate-swaps-instead-of-run-labels.md). GitHub gives a queued job to any idle runner whose labels cover the job's. ADR 0003 let that happen, so a CI runner often ran a job other than the one it was launched for (a swap), and the system repaired the bookkeeping afterwards. Swaps turned out to be routine (the webhook logged three in five seconds on 2026-11-20), and every piece built on top had to allow for them: the swap repair, and #51's re-check, which finds a job's machines by the job-id tag because the runner's name can't be trusted. We now prevent swaps instead. -Every job carries one label unique to it, the **job label**: +Every job must carry one label unique to it, the **job label**: ``` ci-job---- ``` -The launcher registers each runner with its job's labels, so the runner launched for a job carries that job's label, and GitHub can give it no job with another job label. GitHub gives a job to a runner whose labels include all of the job's, so the runner can still take a job of its shape that has no job label: swaps end once every job carries one. The webhook accepts a job label only when it names the job's own run and attempt (`workflow_job.run_id` and `run_attempt`) and has something after them. A job with two, or with one for another run or attempt, is ignored. +The launcher registers each runner with its job's labels, so the runner launched for a job carries that job's label, and GitHub can give it no job with another job label. GitHub gives a job to a runner whose labels include all of the job's, so a runner could still take a job of its shape that has no job label: that is why every job must carry one. The webhook accepts a job label only when it names the job's own run and attempt (`workflow_job.run_id` and `run_attempt`) and has something after them. A job with two, or with one for another run or attempt, is ignored. The rest of the label is the consumer's to make unique within the run. GitHub can't do it for them: inside `runs-on`, `github.job` evaluates to an empty string (measured on 2026-11-20 in a scratch repo, by reading the labels of the queued jobs back from the API), so the job key is written by hand. `strategy.job-index` does work there, and is `0` in a job with no matrix. A reusable workflow sees only its own job keys and its own matrix, so two calls in one run would make the same label: the caller passes something unique into it, as an input. -Rollout is in two steps. First the label is accepted and checked but not required: a job without it is served as before, and swaps are still repaired, including a labelled job's runner taking a job without the label. Once the consumers carry it, the label is required and the swap repair is removed. +Rollout was in two steps. First the label was accepted and checked but not required: a job without it was served as before, and swaps were still repaired, including a labelled job's runner taking a job without the label. Once the consumers carried it, the label became required: a job of our shape without one is ignored, like any other job outside the label contract, and the webhook logs why. The swap repair is removed, and with it the webhook's EC2 and ECS permissions. A runner that runs a job other than its own, which only a job still without the label can make it do, is logged as an anomaly and not repaired; the job it was launched for is relaunched by #51's re-check, which takes a runner busy while its job is still queued as no longer serving it. The re-check also covers runners that never take their job for other reasons (a concurrency group, a failed boot). ## Considered Options diff --git a/docs/runbook.md b/docs/runbook.md index f4d5517..da9adf7 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -352,17 +352,21 @@ $AWS logs tail /aws/lambda/ci-runner-notifier --since 1d **The webhook** logs one JSON line per accepted job, `{"enqueued": "", "job_id": ..., "label": ...}`, and `{"refused": "", "visibility": ...}` -for a job from any repo but a private `zeroae` one (fence 3), and `{"swap": -"", "instance": "i-...", "took": {...}, "requeued": {...}}` (with -`"task": ""` for a Fargate runner) for each swap it repairs: the runner launched for the `requeued` job started the `took` -job instead. Everything else it ignores without a log line. +for a job from any repo but a private `zeroae` one (fence 3), `{"ignored": +"", "job_id": ..., "why": ...}` for a job of our shape with no job label, +or a bad one (see "The job label" in `stack.md`), and `{"anomaly": "runner of +another job", "runner": ..., "launched_for": , "took": {...}}` if a +runner starts a job other than its own, which only a job without a job label +can make it do (`docs/traps.md`). Everything else it ignores without a log line. **The launcher** logs one JSON line per job, carrying the job's fields plus one of: | Field | Meaning | |---|---| -| `"launched": "i-..."`, `"runner"`, `"label"` | an instance is up for the job | +| `"launched": "i-..."`, `"runner"`, `"label"` | an instance is up for the job; with `"swapped": [...]`, because the machines launched for it ran other jobs (a swap, see `docs/traps.md`) | +| `"served": "i-..."` | a machine launched for the job is still up and its runner idle; the job is checked again in 5 minutes | +| `"waiting": "relaunch from