From 64fb04cc6d5b808f31e8bd3fb8ca6f60f625ed51 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patrick=20Sodr=C3=A9?= Date: Mon, 16 Nov 2026 16:18:29 -0500 Subject: [PATCH 1/2] Notifier: name the force-cancel for a lost job GitHub keeps in progress GitHub may keep a job in_progress after its Spot machine is gone: a re-run is refused and a plain cancel ends nothing, so "Re-run the job" alone can't be followed. The reclaim report's summary now gives the run's force-cancel, and warns that it ends the run's other running jobs too. The title is unchanged, since the notifier finds its reports by it. The runbook's "Read a lost job" and the traps table describe the stuck case. Closes #48 Co-Authored-By: Claude Opus 5.5 --- docs/runbook.md | 22 +++++++++++++++++++--- docs/stack.md | 6 +++++- docs/traps.md | 1 + functions/notifier.js | 7 ++++++- test/notifier.test.js | 11 +++++++++++ 5 files changed, 42 insertions(+), 5 deletions(-) diff --git a/docs/runbook.md b/docs/runbook.md index ec7ed1d..f4d5517 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -286,15 +286,31 @@ since GitHub's code search misses private repos. ## Read a lost job -When AWS reclaims a Spot instance or a Fargate Spot task, GitHub marks the job failed, with *The +When AWS reclaims a Spot instance or a Fargate Spot task, GitHub usually marks the job failed, with *The runner has received a shutdown signal*, *lost communication with the server*, or (seen in the live FIS test) just *Process completed with exit code 130* on the running step. On the same commit there is a failed check run from the App, named `ci-runner / ` and titled `Spot capacity reclaimed: re-run this job`. -Its summary names the instance (or the task's id) and links the run. That is -the whole story: re-run the failed job. Nothing retries it automatically. +Its summary names the instance (or the task's id) and links the run. Re-run +the failed job. Nothing retries it automatically. One check run per job: the notifier skips a job whose commit has it already. +GitHub does not always fail the job. On 2026-11-16 an EC2 Spot reclaim left +a job `in_progress`, its step still running, for 15 minutes after the +instance was gone, until the run was force-cancelled. While the job is stuck +like that, `gh run rerun` is refused (`This workflow is already running`) +and `gh run cancel` is accepted but ends nothing. Force-cancel the run, then +re-run its failed jobs: + +```bash +gh api -X POST repos/zeroae//actions/runs//force-cancel +gh run rerun --failed --repo zeroae/ +``` + +A force-cancel ends **every** job still running in the run, the healthy ones +too, so let the run's other jobs finish first. The reclaim check run's +summary gives the force-cancel command with the run's id. + AWS warns 2 minutes before it takes the machine, and a job can finish inside that. When the job ends with `success`, the check run turns `neutral`, titled `The job finished despite the Spot reclaim warning`: no re-run is needed (the diff --git a/docs/stack.md b/docs/stack.md index 4645b2c..8283cce 100644 --- a/docs/stack.md +++ b/docs/stack.md @@ -502,7 +502,11 @@ For each event it: 6. Creates a check run on the head sha: name `ci-runner / ` (or `job ` with no name), `completed`, conclusion `failure`, title `Spot capacity reclaimed: re-run this job`, a summary naming the job and - the instance or task and linking the run, and `details_url` the job's page. + the instance or task, linking the run, and giving the run's force-cancel + (`gh api -X POST repos//actions/runs//force-cancel`) for a job + GitHub keeps in progress (see "Read a lost job" in `docs/runbook.md`), and + `details_url` the job's page. The title must not change: steps 5 and 8 find + a report by it. An EC2 reclaim and a Fargate one give the same check run, but for the summary's machine. 7. Reads the job once more after posting: a job that finished with `success` diff --git a/docs/traps.md b/docs/traps.md index 339d48e..887ef35 100644 --- a/docs/traps.md +++ b/docs/traps.md @@ -15,6 +15,7 @@ Symptoms seen on the stack, and their causes. A quoted section name is in | A job's runner was launched for another job (`$RUNNER_NAME` ends in another job id), and that other job sits "Waiting for a runner" | a swap. The webhook repairs it on the job's `in_progress` (its log says `swap`), and the launcher then launches a runner for the other job. No `swap` line: check the webhook's log for errors (its role needs `ec2:DescribeInstances` and `ec2:CreateTags`, and for a task `ecs:ListTasks`, `ecs:DescribeTasks` and `ecs:TagResource`); cancel and re-run the waiting job | | A Fargate Spot job was lost (its task's `stopCode` is `SpotInterruption`) and its commit has no `ci-runner / ` check run | the notifier logged `ignored: is no job task of this stack` for every change of the stop: it found no job tags on the task. ECS keeps a stopped task's tags readable through `DescribeTasks` (measured live), so check the task's tags (`describe-tasks --include TAGS`, within an hour of the stop; `list-tags-for-resource` refuses a stopped task) and the notifier's role. Re-run the job | | A job fails "The runner has received a shutdown signal", "lost communication" or "exit code 130", next to a failed `ci-runner / ` check | Spot reclaim: re-run the job | +| A job lost to a Spot reclaim stays in progress after its machine is gone; `gh run cancel` does nothing, and `gh run rerun` says `This workflow is already running` | GitHub didn't fail the job its runner died under. Let the run's other jobs finish (a force-cancel ends them all), then `gh api -X POST repos//actions/runs//force-cancel` and re-run the failed jobs (see "Read a lost job") | | A budget lowered for a test is still low after the next `make deploy` | `cloudformation deploy` keeps a parameter's previous value unless it's overridden; restore it with `--parameter-overrides SpotVcpuCap=24 OnDemandVcpuCap=8 FargateSpotVcpuCap=48 FargateVcpuCap=8`. Each draws on its own vCPU quota (see "Deploy and update the stack") | | A job on `ci-fargate-spot` or `ci-fargate` fails at a Docker step: `failed to connect to the docker API at unix:///var/run/docker.sock; check if the path is correct and if the daemon is running`, or a `services:`/`container:` job or Docker action fails to start | Docker used on Fargate. A Fargate capacity has no Docker daemon, by design (ADR-0001): move the job to `ci-spot` or `ci-ondemand`, or drop its Docker use | | The launcher logs `no task launched:` with a reason about a vCPU limit, and the job's message keeps coming back until the dead-letter queue | a Fargate quota at its default (6 vCPUs, `L-36FBB829` Spot, `L-3032A538` On-Demand) or below the capacity's budget. Request an increase (Service Quotas, service `fargate`) or lower `FargateSpotVcpuCap`/`FargateVcpuCap` under the quota; then re-run the job. The quota counts every Fargate task in the account, not only ours | diff --git a/functions/notifier.js b/functions/notifier.js index ba1dcfc..ca0e9ba 100644 --- a/functions/notifier.js +++ b/functions/notifier.js @@ -133,7 +133,12 @@ function createHandler({ secret, fetch, ec2, ecs, now, env, log = console.log }) details_url: `${run}/job/${jobId}`, output: { title: RECLAIMED_TITLE, - summary: `AWS reclaimed the Spot ${machine} running \`${job}\`. Re-run the job: ${run}`, + // GitHub may keep the lost job in_progress after the machine is gone: + // a re-run is then refused and a plain cancel does nothing. + summary: `AWS reclaimed the Spot ${machine} running \`${job}\`. Re-run the job: ${run}\n\n` + + 'If the job still shows in progress, the run cannot be re-run, and a plain cancel does not end it. ' + + `Force-cancel the run, then re-run its failed jobs: \`gh api -X POST repos/${repo}/actions/runs/${t['run-id']}/force-cancel\`. ` + + 'A force-cancel ends every job still running in the run, not only this one: let the others finish first.', }, }); log(`reported ${id}: ${repo} job ${jobId}`); diff --git a/test/notifier.test.js b/test/notifier.test.js index 046ac2a..f7d9c63 100644 --- a/test/notifier.test.js +++ b/test/notifier.test.js @@ -127,6 +127,17 @@ test('an interruption of one of our instances creates a failed check run on the } }); +// GitHub may keep a lost job in_progress after its machine is gone: a re-run is +// then refused and a plain cancel does nothing, so the report names the +// force-cancel, and warns that it ends the run's other running jobs too. +test('a reclaim report tells how to unstick a lost job GitHub keeps in progress', async () => { + const { handler, gh } = setup(); + await handler(EVENT); + const { summary } = gh.calls.find((c) => c.key === 'POST /repos/zeroae/app/check-runs').body.output; + assert.ok(summary.includes('gh api -X POST repos/zeroae/app/actions/runs/444/force-cancel'), summary); + assert.match(summary, /every job still running/); +}); + // A machine is reported only for the job its CI runner is running. The tags // say which job it was launched for, or took in a swap (the webhook re-tags it). for (const [why, job] of [ From 2254696907ddbaffd8695edc794621d3b1290aba Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patrick=20Sodr=C3=A9?= Date: Mon, 16 Nov 2026 16:19:30 -0500 Subject: [PATCH 2/2] Notifier: open the reclaim summary on the failed case; stack.md and traps match it Review of #48: the summary's first instruction now reads "If the job failed, re-run it", so a reader in the stuck case doesn't meet the failing step first. stack.md's step 6 names the re-run and the force-cancel's warning; the traps row uses zeroae/ as the runbook does. Co-Authored-By: Claude Opus 5.5 --- docs/stack.md | 7 ++++--- docs/traps.md | 2 +- functions/notifier.js | 4 ++-- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/stack.md b/docs/stack.md index 8283cce..9a41b5c 100644 --- a/docs/stack.md +++ b/docs/stack.md @@ -502,9 +502,10 @@ For each event it: 6. Creates a check run on the head sha: name `ci-runner / ` (or `job ` with no name), `completed`, conclusion `failure`, title `Spot capacity reclaimed: re-run this job`, a summary naming the job and - the instance or task, linking the run, and giving the run's force-cancel - (`gh api -X POST repos//actions/runs//force-cancel`) for a job - GitHub keeps in progress (see "Read a lost job" in `docs/runbook.md`), and + the instance or task, linking the run, and telling a person to re-run a + failed job or, for a job GitHub keeps in progress, to force-cancel the run + first (its command, with the run's id, and the warning that it ends every + job still running in the run; see "Read a lost job" in `docs/runbook.md`), and `details_url` the job's page. The title must not change: steps 5 and 8 find a report by it. An EC2 reclaim and a Fargate one give the same check run, but for the diff --git a/docs/traps.md b/docs/traps.md index 887ef35..64a78f1 100644 --- a/docs/traps.md +++ b/docs/traps.md @@ -15,7 +15,7 @@ Symptoms seen on the stack, and their causes. A quoted section name is in | A job's runner was launched for another job (`$RUNNER_NAME` ends in another job id), and that other job sits "Waiting for a runner" | a swap. The webhook repairs it on the job's `in_progress` (its log says `swap`), and the launcher then launches a runner for the other job. No `swap` line: check the webhook's log for errors (its role needs `ec2:DescribeInstances` and `ec2:CreateTags`, and for a task `ecs:ListTasks`, `ecs:DescribeTasks` and `ecs:TagResource`); cancel and re-run the waiting job | | A Fargate Spot job was lost (its task's `stopCode` is `SpotInterruption`) and its commit has no `ci-runner / ` check run | the notifier logged `ignored: is no job task of this stack` for every change of the stop: it found no job tags on the task. ECS keeps a stopped task's tags readable through `DescribeTasks` (measured live), so check the task's tags (`describe-tasks --include TAGS`, within an hour of the stop; `list-tags-for-resource` refuses a stopped task) and the notifier's role. Re-run the job | | A job fails "The runner has received a shutdown signal", "lost communication" or "exit code 130", next to a failed `ci-runner / ` check | Spot reclaim: re-run the job | -| A job lost to a Spot reclaim stays in progress after its machine is gone; `gh run cancel` does nothing, and `gh run rerun` says `This workflow is already running` | GitHub didn't fail the job its runner died under. Let the run's other jobs finish (a force-cancel ends them all), then `gh api -X POST repos//actions/runs//force-cancel` and re-run the failed jobs (see "Read a lost job") | +| A job lost to a Spot reclaim stays in progress after its machine is gone; `gh run cancel` does nothing, and `gh run rerun` says `This workflow is already running` | GitHub didn't fail the job its runner died under. Let the run's other jobs finish (a force-cancel ends them all), then `gh api -X POST repos/zeroae//actions/runs//force-cancel` and re-run the failed jobs (see "Read a lost job") | | A budget lowered for a test is still low after the next `make deploy` | `cloudformation deploy` keeps a parameter's previous value unless it's overridden; restore it with `--parameter-overrides SpotVcpuCap=24 OnDemandVcpuCap=8 FargateSpotVcpuCap=48 FargateVcpuCap=8`. Each draws on its own vCPU quota (see "Deploy and update the stack") | | A job on `ci-fargate-spot` or `ci-fargate` fails at a Docker step: `failed to connect to the docker API at unix:///var/run/docker.sock; check if the path is correct and if the daemon is running`, or a `services:`/`container:` job or Docker action fails to start | Docker used on Fargate. A Fargate capacity has no Docker daemon, by design (ADR-0001): move the job to `ci-spot` or `ci-ondemand`, or drop its Docker use | | The launcher logs `no task launched:` with a reason about a vCPU limit, and the job's message keeps coming back until the dead-letter queue | a Fargate quota at its default (6 vCPUs, `L-36FBB829` Spot, `L-3032A538` On-Demand) or below the capacity's budget. Request an increase (Service Quotas, service `fargate`) or lower `FargateSpotVcpuCap`/`FargateVcpuCap` under the quota; then re-run the job. The quota counts every Fargate task in the account, not only ours | diff --git a/functions/notifier.js b/functions/notifier.js index ca0e9ba..7f53404 100644 --- a/functions/notifier.js +++ b/functions/notifier.js @@ -135,10 +135,10 @@ function createHandler({ secret, fetch, ec2, ecs, now, env, log = console.log }) title: RECLAIMED_TITLE, // GitHub may keep the lost job in_progress after the machine is gone: // a re-run is then refused and a plain cancel does nothing. - summary: `AWS reclaimed the Spot ${machine} running \`${job}\`. Re-run the job: ${run}\n\n` + summary: `AWS reclaimed the Spot ${machine} running \`${job}\`. If the job failed, re-run it: ${run}\n\n` + 'If the job still shows in progress, the run cannot be re-run, and a plain cancel does not end it. ' + `Force-cancel the run, then re-run its failed jobs: \`gh api -X POST repos/${repo}/actions/runs/${t['run-id']}/force-cancel\`. ` - + 'A force-cancel ends every job still running in the run, not only this one: let the others finish first.', + + 'A force-cancel ends every job still running in the run: let the others finish first.', }, }); log(`reported ${id}: ${repo} job ${jobId}`);