Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 19 additions & 3 deletions docs/runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -286,15 +286,31 @@ since GitHub's code search misses private repos.

## Read a lost job

When AWS reclaims a Spot instance or a Fargate Spot task, GitHub marks the job failed, with *The
When AWS reclaims a Spot instance or a Fargate Spot task, GitHub usually marks the job failed, with *The
runner has received a shutdown signal*, *lost communication with the server*,
or (seen in the live FIS test) just *Process completed with exit code 130*
on the running step. On the same commit there is a failed check run from the App, named
`ci-runner / <job name>` and titled `Spot capacity reclaimed: re-run this job`.
Its summary names the instance (or the task's id) and links the run. That is
the whole story: re-run the failed job. Nothing retries it automatically.
Its summary names the instance (or the task's id) and links the run. Re-run
the failed job. Nothing retries it automatically.
One check run per job: the notifier skips a job whose commit has it already.

GitHub does not always fail the job. On 2026-11-16 an EC2 Spot reclaim left
a job `in_progress`, its step still running, for 15 minutes after the
instance was gone, until the run was force-cancelled. While the job is stuck
like that, `gh run rerun` is refused (`This workflow is already running`)
and `gh run cancel` is accepted but ends nothing. Force-cancel the run, then
re-run its failed jobs:

```bash
gh api -X POST repos/zeroae/<repo>/actions/runs/<run id>/force-cancel
gh run rerun <run id> --failed --repo zeroae/<repo>
```

A force-cancel ends **every** job still running in the run, the healthy ones
too, so let the run's other jobs finish first. The reclaim check run's
summary gives the force-cancel command with the run's id.

AWS warns 2 minutes before it takes the machine, and a job can finish inside
that. When the job ends with `success`, the check run turns `neutral`, titled
`The job finished despite the Spot reclaim warning`: no re-run is needed (the
Expand Down
7 changes: 6 additions & 1 deletion docs/stack.md
Original file line number Diff line number Diff line change
Expand Up @@ -502,7 +502,12 @@ For each event it:
6. Creates a check run on the head sha: name `ci-runner / <job name>` (or
`job <id>` with no name), `completed`, conclusion `failure`, title
`Spot capacity reclaimed: re-run this job`, a summary naming the job and
the instance or task and linking the run, and `details_url` the job's page.
the instance or task, linking the run, and telling a person to re-run a
failed job or, for a job GitHub keeps in progress, to force-cancel the run
first (its command, with the run's id, and the warning that it ends every
job still running in the run; see "Read a lost job" in `docs/runbook.md`), and
`details_url` the job's page. The title must not change: steps 5 and 8 find
a report by it.
An EC2 reclaim and a Fargate one give the same check run, but for the
summary's machine.
7. Reads the job once more after posting: a job that finished with `success`
Expand Down
1 change: 1 addition & 0 deletions docs/traps.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ Symptoms seen on the stack, and their causes. A quoted section name is in
| A job's runner was launched for another job (`$RUNNER_NAME` ends in another job id), and that other job sits "Waiting for a runner" | a swap. The webhook repairs it on the job's `in_progress` (its log says `swap`), and the launcher then launches a runner for the other job. No `swap` line: check the webhook's log for errors (its role needs `ec2:DescribeInstances` and `ec2:CreateTags`, and for a task `ecs:ListTasks`, `ecs:DescribeTasks` and `ecs:TagResource`); cancel and re-run the waiting job |
| A Fargate Spot job was lost (its task's `stopCode` is `SpotInterruption`) and its commit has no `ci-runner / <job>` check run | the notifier logged `ignored: <task id> is no job task of this stack` for every change of the stop: it found no job tags on the task. ECS keeps a stopped task's tags readable through `DescribeTasks` (measured live), so check the task's tags (`describe-tasks --include TAGS`, within an hour of the stop; `list-tags-for-resource` refuses a stopped task) and the notifier's role. Re-run the job |
| A job fails "The runner has received a shutdown signal", "lost communication" or "exit code 130", next to a failed `ci-runner / <job>` check | Spot reclaim: re-run the job |
| A job lost to a Spot reclaim stays in progress after its machine is gone; `gh run cancel` does nothing, and `gh run rerun` says `This workflow is already running` | GitHub didn't fail the job its runner died under. Let the run's other jobs finish (a force-cancel ends them all), then `gh api -X POST repos/zeroae/<repo>/actions/runs/<run id>/force-cancel` and re-run the failed jobs (see "Read a lost job") |
| A budget lowered for a test is still low after the next `make deploy` | `cloudformation deploy` keeps a parameter's previous value unless it's overridden; restore it with `--parameter-overrides SpotVcpuCap=24 OnDemandVcpuCap=8 FargateSpotVcpuCap=48 FargateVcpuCap=8`. Each draws on its own vCPU quota (see "Deploy and update the stack") |
| A job on `ci-fargate-spot` or `ci-fargate` fails at a Docker step: `failed to connect to the docker API at unix:///var/run/docker.sock; check if the path is correct and if the daemon is running`, or a `services:`/`container:` job or Docker action fails to start | Docker used on Fargate. A Fargate capacity has no Docker daemon, by design (ADR-0001): move the job to `ci-spot` or `ci-ondemand`, or drop its Docker use |
| The launcher logs `no task launched:` with a reason about a vCPU limit, and the job's message keeps coming back until the dead-letter queue | a Fargate quota at its default (6 vCPUs, `L-36FBB829` Spot, `L-3032A538` On-Demand) or below the capacity's budget. Request an increase (Service Quotas, service `fargate`) or lower `FargateSpotVcpuCap`/`FargateVcpuCap` under the quota; then re-run the job. The quota counts every Fargate task in the account, not only ours |
Expand Down
7 changes: 6 additions & 1 deletion functions/notifier.js
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,12 @@ function createHandler({ secret, fetch, ec2, ecs, now, env, log = console.log })
details_url: `${run}/job/${jobId}`,
output: {
title: RECLAIMED_TITLE,
summary: `AWS reclaimed the Spot ${machine} running \`${job}\`. Re-run the job: ${run}`,
// GitHub may keep the lost job in_progress after the machine is gone:
// a re-run is then refused and a plain cancel does nothing.
summary: `AWS reclaimed the Spot ${machine} running \`${job}\`. If the job failed, re-run it: ${run}\n\n`
+ 'If the job still shows in progress, the run cannot be re-run, and a plain cancel does not end it. '
+ `Force-cancel the run, then re-run its failed jobs: \`gh api -X POST repos/${repo}/actions/runs/${t['run-id']}/force-cancel\`. `
+ 'A force-cancel ends every job still running in the run: let the others finish first.',
},
});
log(`reported ${id}: ${repo} job ${jobId}`);
Expand Down
11 changes: 11 additions & 0 deletions test/notifier.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,17 @@ test('an interruption of one of our instances creates a failed check run on the
}
});

// GitHub may keep a lost job in_progress after its machine is gone: a re-run is
// then refused and a plain cancel does nothing, so the report names the
// force-cancel, and warns that it ends the run's other running jobs too.
test('a reclaim report tells how to unstick a lost job GitHub keeps in progress', async () => {
const { handler, gh } = setup();
await handler(EVENT);
const { summary } = gh.calls.find((c) => c.key === 'POST /repos/zeroae/app/check-runs').body.output;
assert.ok(summary.includes('gh api -X POST repos/zeroae/app/actions/runs/444/force-cancel'), summary);
assert.match(summary, /every job still running/);
});

// A machine is reported only for the job its CI runner is running. The tags
// say which job it was launched for, or took in a swap (the webhook re-tags it).
for (const [why, job] of [
Expand Down
Loading