From 02f47a55fb141978816f2c85fa99055f45b9c6d5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patrick=20Sodr=C3=A9?= Date: Fri, 20 Nov 2026 00:36:31 -0500 Subject: [PATCH 1/4] Launcher: a queued job is checked again, so a runner that never takes it is replaced A runner launched for a queued job can idle out (15 minutes) before the job can run, for example while a job-level concurrency group is held by another job. Nothing checked the job again, so it stayed queued with no runner. Every launch, and every check that finds the job still queued and served, now sends the job's own message again, 300 s on. A check looks for a live instance or task tagged with the job id: if one serves it, the check goes on; if none does, the job is launched. The check ends when GitHub no longer has the job queued. Its one listing of the stack's machines also feeds the vCPU budget. Co-Authored-By: Claude Haiku 5.5 --- docs/stack.md | 17 +++++++- docs/traps.md | 2 +- functions/launcher.js | 56 ++++++++++++++++-------- test/launcher-ecs.test.js | 5 ++- test/launcher.test.js | 92 +++++++++++++++++++++++++++++++++++++-- test/tasks-ecs.test.js | 2 +- 6 files changed, 148 insertions(+), 26 deletions(-) diff --git a/docs/stack.md b/docs/stack.md index 9a41b5c..3e7fb65 100644 --- a/docs/stack.md +++ b/docs/stack.md @@ -318,8 +318,21 @@ failures per message), for each job: 7. If no instance or task launches, it removes the registration it just minted and fails the message, which comes back 120 s later. -There is no de-duplication on job id. A duplicate delivery launches a second -runner at worst, and the idle exit and the sweeper remove it. +**The check that follows (#51).** Every launch, and every check that finds the +job still queued and served, sends the job's own message again, delayed 300 s +(`RECHECK_SECONDS`), once the launch is done. A check first looks at the +machines this stack tags with the job (`ci-runner:job-id`): a `pending` or +`running` instance, or a task not yet told to stop. If one is serving the job, +the check sends its own message on and launches nothing (its log line says +`served`). If none is, the job is launched as above. The checks end when GitHub +no longer has the job queued, which drops the message. This is why a job held +behind a concurrency group the first job holds gets a runner once the group +frees: a runner that idled out first is replaced at the next check. The sweeper +does not do this, since it has no list of queued jobs. Not yet checked live: +the behaviour is covered with fakes in `test/launcher.test.js`. + +There is no de-duplication on job id beyond that check. A duplicate delivery +launches a second runner at worst, and the idle exit and the sweeper remove it. `cloudformation deploy` keeps a parameter's previous value unless it's overridden, so after lowering a budget for a test, restore it explicitly diff --git a/docs/traps.md b/docs/traps.md index 64a78f1..fcb6f42 100644 --- a/docs/traps.md +++ b/docs/traps.md @@ -9,7 +9,7 @@ Symptoms seen on the stack, and their causes. A quoted section name is in | `permission denied` on `/var/run/docker.sock` | the host's `docker` group isn't gid 123 | | `setup-python` fails, or picks up a broken Python | `AGENT_TOOLSDIRECTORY` isn't `/opt/hostedtoolcache` | | A deploy waits 60 minutes, then fails on the state lock | an `apply` ran on Spot and was reclaimed; it must be `ci-ondemand` | -| A job sits "Waiting for a runner" | its capacity's vCPU budget is full, the repo isn't in the group, the App isn't installed on it, or the repo is public. Also check the label contract: `self-hosted`, exactly one of `ci-spot`/`ci-ondemand`/`ci-fargate-spot`/`ci-fargate`, exactly one size label (`ci-cpu-2`/`-4`/`-8`/`-16`), exactly one of `x64`/`arm64`, and otherwise only `linux`. The launcher's log tells the first case (`deferred`) apart from the rest (no line at all). A job that queued before the repo was turned on stays waiting after the fix: cancel and re-run it | +| A job sits "Waiting for a runner" | its capacity's vCPU budget is full, the repo isn't in the group, the App isn't installed on it, or the repo is public. Also check the label contract: `self-hosted`, exactly one of `ci-spot`/`ci-ondemand`/`ci-fargate-spot`/`ci-fargate`, exactly one size label (`ci-cpu-2`/`-4`/`-8`/`-16`), exactly one of `x64`/`arm64`, and otherwise only `linux`. The launcher's log tells the first case (`deferred`) apart from the rest (no line at all). A job held behind a concurrency group shows `served` while its runner is up; once that runner idles out, the job's next check (5 minutes apart, see "Launching for queued jobs" in `stack.md`) launches another. A job that queued before the repo was turned on stays waiting after the fix: cancel and re-run it | | A job that ran on our runners before now sits "Waiting for a runner", and the webhook's log shows no `enqueued` line for it | missing size or architecture label: its `runs-on` names only its capacity (`self-hosted`, `ci-spot`). There are no defaults; add a size and an architecture label (`ci-cpu-4` and `x64` keep the old shape), then cancel and re-run the job | | An `arm64` runner's instance comes up and terminates without taking its job, and its console shows `no matching manifest for linux/arm64` | `ghcr.io/zeroae/ci-runner:latest` has no arm64 entry yet: the image workflow hasn't pushed a multi-arch `:latest` since it began building arm64. Run the image workflow on `main`, then re-run the job | | A job's runner was launched for another job (`$RUNNER_NAME` ends in another job id), and that other job sits "Waiting for a runner" | a swap. The webhook repairs it on the job's `in_progress` (its log says `swap`), and the launcher then launches a runner for the other job. No `swap` line: check the webhook's log for errors (its role needs `ec2:DescribeInstances` and `ec2:CreateTags`, and for a task `ecs:ListTasks`, `ecs:DescribeTasks` and `ecs:TagResource`); cancel and re-run the waiting job | diff --git a/functions/launcher.js b/functions/launcher.js index 47f3a2b..b5a6df6 100644 --- a/functions/launcher.js +++ b/functions/launcher.js @@ -26,6 +26,11 @@ const ecsTag = ([key, v]) => ({ key, value: String(v).replace(ECS_TAG_CHARS, '_' // without the ci-runner: prefix. The webhook and the notifier carry copies. const tagsOf = (tags) => Object.fromEntries((tags || []).map((x) => [(x.Key ?? x.key).replace(/^ci-runner:/, ''), x.Value ?? x.value])); +// A queued job is checked again this many seconds after each check (#51): a +// runner that idled out, or never got the job, is replaced by the next check. +// SQS allows a delay of up to 900 seconds. +const RECHECK_SECONDS = 300; + const b64u = (s) => Buffer.from(s).toString('base64url'); function appJwt(appId, pem, nowMs) { @@ -50,6 +55,8 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l const org = env.GITHUB_ORG; const subnets = env.SUBNET_IDS.split(','); const budgets = { spot: env.SPOT_VCPU_CAP, 'on-demand': env.ONDEMAND_VCPU_CAP, 'fargate-spot': env.FARGATE_SPOT_VCPU_CAP, fargate: env.FARGATE_VCPU_CAP }; + // The job's own message, delayed: its next check (#51). + const checkLater = (m) => ({ QueueUrl: env.QUEUE_URL, MessageBody: JSON.stringify(m), DelaySeconds: RECHECK_SECONDS }); const tokens = new Map(); // installation id -> { token, until } let app; let groupId; @@ -87,12 +94,13 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l } } - // vCPUs of this stack's live instances on one capacity, from the shape tags. - // An instance launched before them has no tags: it has the default shape's - // vCPUs, and its capacity is Spot only with the Spot marker (On-Demand has - // no InstanceLifecycle). - async function vcpusIn(type) { - let n = 0; + // This stack's instances that are not terminated, one record each: its job + // (the ci-runner:job-id tag), its shape from the shape tags, and whether it + // is still serving that job (pending or running). An instance launched before + // the shape tags has the default shape, and its capacity is Spot only with + // the Spot marker (On-Demand has no InstanceLifecycle). + async function instancesLive() { + const out = []; let NextToken; do { const r = await ec2.describeInstances({ @@ -106,12 +114,12 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l for (const i of res.Instances || []) { const t = tagsOf(i.Tags); const capacity = CAPACITY[t.capacity] || (i.InstanceLifecycle === 'spot' ? 'spot' : 'on-demand'); - if (capacity === type) n += Number(t.vcpus) || DEFAULT_SHAPE.vcpus; + out.push({ id: i.InstanceId, job: t['job-id'], capacity, vcpus: Number(t.vcpus) || DEFAULT_SHAPE.vcpus, serving: ['pending', 'running'].includes(i.State.Name) }); } } NextToken = r.NextToken; } while (NextToken); - return n; + return out; } // The tags a runner's instance or task carries: the job, for the swap @@ -162,11 +170,11 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l return arn; } - // vCPUs of the cluster's tasks on one Fargate capacity, from the shape - // tags. A task counts until it is STOPPED: one told to stop (desired - // STOPPED) still holds its vCPUs while it deprovisions. - async function taskVcpusIn(type) { - let n = 0; + // The cluster's tasks, one record each, as instancesLive. A task holds its + // vCPUs until it is STOPPED: one told to stop (desired STOPPED) still holds + // them while it deprovisions, but is no longer serving its job. + async function tasksLive() { + const out = []; for (const desiredStatus of ['RUNNING', 'STOPPED']) { let nextToken; do { @@ -175,13 +183,14 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l const d = await ecs.describeTasks({ cluster: env.CLUSTER, tasks: r.taskArns, include: ['TAGS'] }); for (const task of d.tasks || []) { const t = tagsOf(task.tags); - if (task.lastStatus !== 'STOPPED' && CAPACITY[t.capacity] === type) n += Number(t.vcpus) || DEFAULT_SHAPE.vcpus; + if (task.lastStatus === 'STOPPED') continue; + out.push({ id: task.taskArn, job: t['job-id'], capacity: CAPACITY[t.capacity], vcpus: Number(t.vcpus) || DEFAULT_SHAPE.vcpus, serving: desiredStatus === 'RUNNING' }); } } nextToken = r.nextToken; } while (nextToken); } - return n; + return out; } async function launchInstance(m, name, runnerId, jitConfig) { @@ -229,12 +238,22 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l if (!j) return log(JSON.stringify({ ...job, dropped: 'job not found' })); if (j.status !== 'queued') return log(JSON.stringify({ ...job, dropped: j.status })); + // One listing serves both checks. A job whose runner is still serving it is + // not launched again: its check goes on, so a runner that idles out without + // the job is replaced by a later check (#51). + const type = CAPACITY[m.label]; + const machines = await (PROVIDER[m.label] ? tasksLive() : instancesLive()); + const served = machines.find((x) => x.serving && x.job === String(m.job_id)); + if (served) { + await sqs.sendMessage(checkLater(m)); + return log(JSON.stringify({ ...job, served: served.id })); + } + // A soft budget: launches checked at the same moment may overshoot it. A // job that would go over it goes back on the queue as a new message, so // retries don't pile up towards the dead-letter queue. - const type = CAPACITY[m.label]; const budget = Number(budgets[type]); - const held = await (PROVIDER[m.label] ? taskVcpusIn(type) : vcpusIn(type)); + const held = machines.filter((x) => x.capacity === type).reduce((n, x) => n + x.vcpus, 0); const { vcpus } = shapeOf(m); if (held + vcpus > budget) { await sqs.sendMessage({ QueueUrl: env.QUEUE_URL, MessageBody: JSON.stringify(m), DelaySeconds: 60 }); @@ -258,6 +277,9 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l await gh('DELETE', `/orgs/${org}/actions/runners/${r.runner.id}`, token).catch(() => {}); throw e; } + // After the launch, so a failed send here never removes a live runner: the + // retry then finds it served and sends the check. + await sqs.sendMessage(checkLater(m)); } return async (event) => { diff --git a/test/launcher-ecs.test.js b/test/launcher-ecs.test.js index 2b7d290..418d12d 100644 --- a/test/launcher-ecs.test.js +++ b/test/launcher-ecs.test.js @@ -73,12 +73,15 @@ const job = (label, vcpus, arch, extra = {}) => ({ Records: [{ messageId: 'm0', }) }] }); test('ECS accepts the launcher\'s RunTask for every Fargate capacity, size and architecture', { skip }, async () => { + // One job id per launch: a task still serving a job is not launched again (#51), + // and the emulator keeps the tasks of earlier launches. + let jobId = 1000; for (const [label, provider] of [['ci-fargate-spot', 'FARGATE_SPOT'], ['ci-fargate', 'FARGATE']]) { for (const vcpus of [2, 4, 8, 16]) { for (const arch of ['x64', 'arm64']) { const what = `${label} ${vcpus} ${arch}`; const { handler, logs } = launcher(fakeFetch()); - assert.deepEqual(await handler(job(label, vcpus, arch)), { batchItemFailures: [] }, `${what}: ${logs.join('\n')}`); + assert.deepEqual(await handler(job(label, vcpus, arch, { job_id: jobId++ })), { batchItemFailures: [] }, `${what}: ${logs.join('\n')}`); const arn = JSON.parse(logs.find((l) => l.includes('"launched"'))).launched; const [task] = (await ecs.describeTasks({ cluster: 'ci-runner', tasks: [arn], include: ['TAGS'] })).tasks; assert.equal(task.capacityProviderName, provider, what); diff --git a/test/launcher.test.js b/test/launcher.test.js index 8dfbb4c..fcf1cba 100644 --- a/test/launcher.test.js +++ b/test/launcher.test.js @@ -62,10 +62,11 @@ function fakeEc2({ fleet = { Instances: [{ InstanceIds: ['i-0123'] }], Errors: [ const calls = []; return { calls, + running, describeInstances: async (input) => { calls.push(['describe', input]); const all = running.map((r, i) => ({ InstanceId: `i-r${i}`, State: { Name: r.state || 'running' }, ...(r.lifecycle ? { InstanceLifecycle: r.lifecycle } : {}), - Tags: [['ci-runner:stack', 'ci-runner'], ['ci-runner:capacity', r.capacity], ['ci-runner:vcpus', r.vcpus], ['ci-runner:arch', r.arch]] + Tags: [['ci-runner:stack', 'ci-runner'], ['ci-runner:capacity', r.capacity], ['ci-runner:vcpus', r.vcpus], ['ci-runner:arch', r.arch], ['ci-runner:job-id', r.job]] .filter(([, v]) => v !== undefined).map(([Key, Value]) => ({ Key, Value: String(Value) })) })); // Two pages, to check NextToken is followed. if (!input.NextToken) return { Reservations: [{ Instances: all.slice(0, 1) }], NextToken: all.length > 1 ? 'p2' : undefined }; @@ -84,7 +85,7 @@ function fakeEcs({ tasks = [], run } = {}) { const calls = []; const all = tasks.map((t, i) => ({ taskArn: `arn:aws:ecs:us-east-1:1:task/ci-runner/t${i}`, desiredStatus: t.desired || 'RUNNING', lastStatus: t.last || 'RUNNING', - tags: [['ci-runner:stack', 'ci-runner'], ['ci-runner:capacity', t.capacity], ['ci-runner:vcpus', t.vcpus]] + tags: [['ci-runner:stack', 'ci-runner'], ['ci-runner:capacity', t.capacity], ['ci-runner:vcpus', t.vcpus], ['ci-runner:job-id', t.job]] .filter(([, v]) => v !== undefined).map(([key, value]) => ({ key, value: String(value) })), })); return { @@ -306,6 +307,11 @@ const OLD_SPOT = { lifecycle: 'spot' }; const OLD_OD = {}; const OD_MSG = { ...MSG, label: 'ci-ondemand', labels: ['self-hosted', 'ci-ondemand'] }; const launched = (ec2) => ec2.calls.some((c) => c[0] === 'createFleet'); +// The check a queued job schedules for itself (#51): the same message, later. +const RECHECK = 300; +const checkLater = (m) => ({ QueueUrl: ENV.QUEUE_URL, MessageBody: JSON.stringify(m), DelaySeconds: RECHECK }); +const fleets = (ec2) => ec2.calls.filter((c) => c[0] === 'createFleet').length; +const again = (sqs, i) => JSON.parse(sqs[i].MessageBody); test('the budget counts the stack\'s live instances by tag', async () => { const { handler, ec2 } = setup(); @@ -323,7 +329,7 @@ test('under the Spot budget, mixed sizes of both architectures leave room: the j const { handler, sqs } = setup({ ec2 }); await handler(sqsEvent({ ...MSG, vcpus: 8 })); assert.ok(launched(ec2)); - assert.deepEqual(sqs, []); + assert.deepEqual(sqs, [checkLater({ ...MSG, vcpus: 8 })]); }); test('a job that brings its capacity exactly to the budget launches', async () => { @@ -332,7 +338,7 @@ test('a job that brings its capacity exactly to the budget launches', async () = const { handler, sqs } = setup({ ec2 }); await handler(sqsEvent(MSG)); assert.ok(launched(ec2)); - assert.deepEqual(sqs, []); + assert.deepEqual(sqs, [checkLater(MSG)]); }); test('a job that would go over the Spot budget goes back on the queue with a delay, logged in vCPUs', async () => { @@ -572,3 +578,81 @@ test('a failed RunTask removes the registration and fails the message, as a fail assert.match(err, run === failed ? /^no task launched: Capacity is unavailable at this time: FARGATE_SPOT$/ : /AccessDenied/); } }); + +// --- a queued job keeps its check (#51) ------------------------------------ +// A launch's runner can go without taking the job (it idles out, or the job +// waits behind a concurrency group the first job holds). So each queued job +// checks again later, with its own message: a live runner tagged with the job +// is served and the check repeats; none means a launch. + +test('a launch schedules its own check: the same message, five minutes on', async () => { + const { handler, sqs } = setup(); + await handler(sqsEvent(MSG)); + assert.deepEqual(sqs, [checkLater(MSG)]); +}); + +test('a queued job whose launch produced no runner gets a new launch at its check', async () => { + const ec2 = fakeEc2(); // the launch left no instance behind + const { handler, sqs } = setup({ ec2 }); + await handler(sqsEvent(MSG)); + assert.equal(fleets(ec2), 1); + assert.deepEqual(await handler(sqsEvent(again(sqs, 0))), { batchItemFailures: [] }); + assert.equal(fleets(ec2), 2, 'launched again'); + assert.deepEqual(sqs[1], checkLater(MSG), 'and checks again'); +}); + +test('a job held behind a concurrency group keeps its runner while it is up, and is launched again once that runner has gone', async () => { + const running = []; + const ec2 = fakeEc2({ running }); + const { handler, sqs } = setup({ ec2 }); + await handler(sqsEvent(MSG)); + assert.equal(fleets(ec2), 1); + running.push({ job: MSG.job_id, capacity: 'spot', vcpus: 4, arch: 'x64', lifecycle: 'spot' }); // idle, group still held + await handler(sqsEvent(again(sqs, 0))); + assert.equal(fleets(ec2), 1, 'a live runner for the job is not replaced'); + assert.deepEqual(sqs[1], checkLater(MSG), 'the check goes on'); + running[0].state = 'terminated'; // idled out before the group freed + await handler(sqsEvent(again(sqs, 1))); + assert.equal(fleets(ec2), 2, 'with no runner left, the job is launched again'); +}); + +test('a check for a job that has started or finished launches nothing and schedules no further check', async () => { + for (const jobStatus of ['in_progress', 'completed']) { + const ec2 = fakeEc2(); + const { handler, sqs } = setup({ gh: fakeGitHub({ jobStatus }), ec2 }); + assert.deepEqual(await handler(sqsEvent(MSG)), { batchItemFailures: [] }); + assert.equal(fleets(ec2), 0, jobStatus); + assert.deepEqual(sqs, [], jobStatus); + } +}); + +test('two checks in a row launch one runner, not two', async () => { + const running = []; + const ec2 = fakeEc2({ running }); + const { handler, sqs } = setup({ ec2 }); + await handler(sqsEvent(MSG)); + running.push({ job: MSG.job_id, capacity: 'spot', vcpus: 4, arch: 'x64', lifecycle: 'spot' }); // its runner is up + await handler(sqsEvent(MSG)); // a repeat delivery of the original + await handler(sqsEvent(again(sqs, 0))); + assert.equal(fleets(ec2), 1); +}); + +test('a check of a message from before the size labels launches the default shape, and checks again', async () => { + const old = { ...MSG }; + delete old.vcpus; + delete old.arch; + const ec2 = fakeEc2(); + const { handler, sqs } = setup({ ec2 }); + await handler(sqsEvent(old)); + await handler(sqsEvent(again(sqs, 0))); + assert.equal(fleets(ec2), 2); + assert.deepEqual(sqs[1], checkLater(old)); +}); + +test('a Fargate job served by its own task is not launched again, and its check goes on', async () => { + const ecs = fakeEcs({ tasks: [{ capacity: 'fargate-spot', vcpus: 4, job: FG_MSG.job_id }] }); + const { handler, sqs } = setup({ ecs }); + assert.deepEqual(await handler(sqsEvent(FG_MSG)), { batchItemFailures: [] }); + assert.equal(runTaskInput(ecs), undefined, 'no RunTask'); + assert.deepEqual(sqs, [checkLater(FG_MSG)]); +}); diff --git a/test/tasks-ecs.test.js b/test/tasks-ecs.test.js index 679965d..11a7395 100644 --- a/test/tasks-ecs.test.js +++ b/test/tasks-ecs.test.js @@ -118,7 +118,7 @@ function fakeGitHub(runners = [], job = {}) { async function launch() { const logs = []; const h = launcher.createHandler({ - secret, fetch: fakeGitHub().fetch, ecs, ec2: {}, sqs: {}, now: Date.now, env, log: (l) => logs.push(l), + secret, fetch: fakeGitHub().fetch, ecs, ec2: {}, sqs: { sendMessage: async () => ({}) }, now: Date.now, env, log: (l) => logs.push(l), }); const body = JSON.stringify({ repo: 'zeroae/lib', job_id: 999, run_id: 888, head_sha: 'fff000', job_name: JOB_NAMES[999], From 4a79daf00ddee529450516d83ba222a12646f32b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patrick=20Sodr=C3=A9?= Date: Fri, 20 Nov 2026 00:38:05 -0500 Subject: [PATCH 2/4] Launcher tests: the job's status changes under the concurrency group, and cancelled jobs The fake GitHub's job status can now change within a test. The concurrency test ends with the group freeing: the job is taken and in progress, and its check schedules nothing. The started-or-finished guard also runs with a live runner tagged with the job. GitHub reports a cancelled job as status "completed" (probed on zeroae/ci-runner, run 43221273888, 2026-11-20). Co-Authored-By: Claude Haiku 5.5 --- test/launcher.test.js | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/test/launcher.test.js b/test/launcher.test.js index fcf1cba..ace9905 100644 --- a/test/launcher.test.js +++ b/test/launcher.test.js @@ -33,6 +33,7 @@ function verifyJwt(auth) { // A fake GitHub: routes keyed "METHOD path". Records every call. function fakeGitHub({ jobStatus = 'queued', jobHttp = 200, jit = { status: 201 }, groups = [{ id: 1, name: 'Default' }, { id: 3, name: 'ci-runner' }] } = {}) { const calls = []; + const job = { status: jobStatus }; // a test can change it, as GitHub does const fetch = async (url, init = {}) => { const u = new URL(url); const key = `${init.method || 'GET'} ${u.pathname}`; @@ -45,7 +46,7 @@ function fakeGitHub({ jobStatus = 'queued', jobHttp = 200, jit = { status: 201 } return res(201, { token: 'ghs_install', expires_at: new Date(T0 + 3600e3).toISOString() }); } assert.equal(init.headers.authorization, 'Bearer ghs_install', key); - if (key === 'GET /repos/zeroae/app/actions/jobs/555') return res(jobHttp, jobHttp === 200 ? { id: 555, status: jobStatus } : { message: 'x' }); + if (key === 'GET /repos/zeroae/app/actions/jobs/555') return res(jobHttp, jobHttp === 200 ? { id: 555, status: job.status } : { message: 'x' }); if (key === 'GET /orgs/zeroae/actions/runner-groups') return res(200, { total_count: groups.length, runner_groups: groups }); if (key === 'POST /orgs/zeroae/actions/runners/generate-jitconfig') { return jit.status === 201 ? res(201, { runner: { id: 77, name: body.name }, encoded_jit_config: JIT }) : res(jit.status, { message: 'nope' }); @@ -53,7 +54,7 @@ function fakeGitHub({ jobStatus = 'queued', jobHttp = 200, jit = { status: 201 } if (key === 'DELETE /orgs/zeroae/actions/runners/77') return res(204, null); throw new Error('unexpected ' + key); }; - return { fetch, calls, keys: () => calls.map((c) => c.key) }; + return { fetch, calls, job, keys: () => calls.map((c) => c.key) }; } // `running` lists the stack's instances: { lifecycle: 'spot' | undefined, state, @@ -604,8 +605,8 @@ test('a queued job whose launch produced no runner gets a new launch at its chec test('a job held behind a concurrency group keeps its runner while it is up, and is launched again once that runner has gone', async () => { const running = []; const ec2 = fakeEc2({ running }); - const { handler, sqs } = setup({ ec2 }); - await handler(sqsEvent(MSG)); + const { handler, sqs, gh } = setup({ ec2 }); + await handler(sqsEvent(MSG)); // the job is queued behind the held group assert.equal(fleets(ec2), 1); running.push({ job: MSG.job_id, capacity: 'spot', vcpus: 4, arch: 'x64', lifecycle: 'spot' }); // idle, group still held await handler(sqsEvent(again(sqs, 0))); @@ -614,11 +615,20 @@ test('a job held behind a concurrency group keeps its runner while it is up, and running[0].state = 'terminated'; // idled out before the group freed await handler(sqsEvent(again(sqs, 1))); assert.equal(fleets(ec2), 2, 'with no runner left, the job is launched again'); -}); - -test('a check for a job that has started or finished launches nothing and schedules no further check', async () => { + // The group frees: the job is taken by that runner, so it is in progress, + // and the next check drops the job and schedules nothing. + gh.job.status = 'in_progress'; + await handler(sqsEvent(again(sqs, 2))); + assert.equal(fleets(ec2), 2, 'not launched again'); + assert.equal(sqs.length, 3, 'and no further check'); +}); + +// GitHub reports a cancelled job as status "completed" with conclusion +// "cancelled" (probed on zeroae/ci-runner, run 43221273888, 2026-11-20), so +// "completed" covers the cancelled case too. +test('a check for a job that has started, finished or was cancelled launches nothing and schedules no further check', async () => { for (const jobStatus of ['in_progress', 'completed']) { - const ec2 = fakeEc2(); + const ec2 = fakeEc2({ running: [{ job: MSG.job_id, capacity: 'spot', vcpus: 4, arch: 'x64', lifecycle: 'spot' }] }); const { handler, sqs } = setup({ gh: fakeGitHub({ jobStatus }), ec2 }); assert.deepEqual(await handler(sqsEvent(MSG)), { batchItemFailures: [] }); assert.equal(fleets(ec2), 0, jobStatus); From a96a0df141b7e2637c957a1316a592ff61fecde9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patrick=20Sodr=C3=A9?= Date: Fri, 20 Nov 2026 00:53:13 -0500 Subject: [PATCH 3/4] Launcher: relaunches of a job its runners never take back off to an hour The owner widened #51: a job whose runners never take it (the repo left the group, labels nothing matches) was relaunched about every 15 to 20 minutes for as long as GitHub kept it queued. A relaunch now waits at least 15 minutes after the job's last launch, the gap doubling for each launch after that, up to 60 minutes. A held concurrency group still frees into a runner within the hour, so the job is never given up. The message carries `launches` and `launched_at`; one without them (the webhook's, a swap's re-queue, a message from before this change) launches at once. Co-Authored-By: Claude Opus 5.5 --- docs/stack.md | 8 +++++- docs/traps.md | 2 +- functions/launcher.js | 16 ++++++++++- test/launcher.test.js | 64 ++++++++++++++++++++++++++++++++++--------- 4 files changed, 74 insertions(+), 16 deletions(-) diff --git a/docs/stack.md b/docs/stack.md index 3e7fb65..ad23fcc 100644 --- a/docs/stack.md +++ b/docs/stack.md @@ -324,7 +324,13 @@ job still queued and served, sends the job's own message again, delayed 300 s machines this stack tags with the job (`ci-runner:job-id`): a `pending` or `running` instance, or a task not yet told to stop. If one is serving the job, the check sends its own message on and launches nothing (its log line says -`served`). If none is, the job is launched as above. The checks end when GitHub +`served`). If none is, the job is launched as above, but no sooner than 15 +minutes after its last launch, a gap that doubles for each launch after that, up +to 60 minutes (its log line says `waiting` until then). So a job whose runners +never take it (the repo left the group, labels nothing matches) costs about one +launch an hour, not one every check. The message carries `launches` and +`launched_at` for this; one without them (the webhook's, a swap's re-queue) has +had no launch and launches at once. The checks end when GitHub no longer has the job queued, which drops the message. This is why a job held behind a concurrency group the first job holds gets a runner once the group frees: a runner that idled out first is replaced at the next check. The sweeper diff --git a/docs/traps.md b/docs/traps.md index fcb6f42..4bfde87 100644 --- a/docs/traps.md +++ b/docs/traps.md @@ -9,7 +9,7 @@ Symptoms seen on the stack, and their causes. A quoted section name is in | `permission denied` on `/var/run/docker.sock` | the host's `docker` group isn't gid 123 | | `setup-python` fails, or picks up a broken Python | `AGENT_TOOLSDIRECTORY` isn't `/opt/hostedtoolcache` | | A deploy waits 60 minutes, then fails on the state lock | an `apply` ran on Spot and was reclaimed; it must be `ci-ondemand` | -| A job sits "Waiting for a runner" | its capacity's vCPU budget is full, the repo isn't in the group, the App isn't installed on it, or the repo is public. Also check the label contract: `self-hosted`, exactly one of `ci-spot`/`ci-ondemand`/`ci-fargate-spot`/`ci-fargate`, exactly one size label (`ci-cpu-2`/`-4`/`-8`/`-16`), exactly one of `x64`/`arm64`, and otherwise only `linux`. The launcher's log tells the first case (`deferred`) apart from the rest (no line at all). A job held behind a concurrency group shows `served` while its runner is up; once that runner idles out, the job's next check (5 minutes apart, see "Launching for queued jobs" in `stack.md`) launches another. A job that queued before the repo was turned on stays waiting after the fix: cancel and re-run it | +| A job sits "Waiting for a runner" | its capacity's vCPU budget is full, the repo isn't in the group, the App isn't installed on it, or the repo is public. Also check the label contract: `self-hosted`, exactly one of `ci-spot`/`ci-ondemand`/`ci-fargate-spot`/`ci-fargate`, exactly one size label (`ci-cpu-2`/`-4`/`-8`/`-16`), exactly one of `x64`/`arm64`, and otherwise only `linux`. The launcher's log tells the first case (`deferred`) apart from the rest (no line at all). A job held behind a concurrency group shows `served` while its runner is up; once that runner idles out, a later check (5 minutes apart, see "Launching for queued jobs" in `stack.md`) launches another, no sooner than 15 minutes after the last launch and backing off to an hour (`waiting` in the log). A job that queued before the repo was turned on stays waiting after the fix: cancel and re-run it | | A job that ran on our runners before now sits "Waiting for a runner", and the webhook's log shows no `enqueued` line for it | missing size or architecture label: its `runs-on` names only its capacity (`self-hosted`, `ci-spot`). There are no defaults; add a size and an architecture label (`ci-cpu-4` and `x64` keep the old shape), then cancel and re-run the job | | An `arm64` runner's instance comes up and terminates without taking its job, and its console shows `no matching manifest for linux/arm64` | `ghcr.io/zeroae/ci-runner:latest` has no arm64 entry yet: the image workflow hasn't pushed a multi-arch `:latest` since it began building arm64. Run the image workflow on `main`, then re-run the job | | A job's runner was launched for another job (`$RUNNER_NAME` ends in another job id), and that other job sits "Waiting for a runner" | a swap. The webhook repairs it on the job's `in_progress` (its log says `swap`), and the launcher then launches a runner for the other job. No `swap` line: check the webhook's log for errors (its role needs `ec2:DescribeInstances` and `ec2:CreateTags`, and for a task `ecs:ListTasks`, `ecs:DescribeTasks` and `ecs:TagResource`); cancel and re-run the waiting job | diff --git a/functions/launcher.js b/functions/launcher.js index b5a6df6..3971c8c 100644 --- a/functions/launcher.js +++ b/functions/launcher.js @@ -30,6 +30,13 @@ const tagsOf = (tags) => Object.fromEntries((tags || []).map((x) => [(x.Key ?? x // runner that idled out, or never got the job, is replaced by the next check. // SQS allows a delay of up to 900 seconds. const RECHECK_SECONDS = 300; +// A job whose runners never take it (the repo left the group, labels nothing +// matches) is relaunched no sooner than this after its last launch: 15 minutes, +// a runner's idle exit, for the first relaunch, doubling for each after, up to +// 60 minutes. A held concurrency group still frees into a runner within the +// hour. The message carries the count and the time of the job's last launch; +// one without them (the webhook's, a swap's re-queue) has had none. +const relaunchGapMs = (launches) => Math.min(15 * 2 ** (launches - 1), 60) * 60e3; const b64u = (s) => Buffer.from(s).toString('base64url'); @@ -249,6 +256,13 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l return log(JSON.stringify({ ...job, served: served.id })); } + // Launched before and not served now: wait out the gap before a relaunch. + const launches = Number.isInteger(m.launches) && m.launches > 0 ? m.launches : 0; + if (launches && now() - m.launched_at < relaunchGapMs(launches)) { + await sqs.sendMessage(checkLater(m)); + return log(JSON.stringify({ ...job, waiting: `relaunch ${launches + 1} from ${new Date(m.launched_at + relaunchGapMs(launches)).toISOString()}` })); + } + // A soft budget: launches checked at the same moment may overshoot it. A // job that would go over it goes back on the queue as a new message, so // retries don't pile up towards the dead-letter queue. @@ -279,7 +293,7 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l } // After the launch, so a failed send here never removes a live runner: the // retry then finds it served and sends the check. - await sqs.sendMessage(checkLater(m)); + await sqs.sendMessage(checkLater({ ...m, launches: launches + 1, launched_at: now() })); } return async (event) => { diff --git a/test/launcher.test.js b/test/launcher.test.js index ace9905..5e134c6 100644 --- a/test/launcher.test.js +++ b/test/launcher.test.js @@ -112,13 +112,13 @@ function fakeEcs({ tasks = [], run } = {}) { }; } -function setup({ gh = fakeGitHub(), ec2 = fakeEc2(), ecs = fakeEcs(), env = ENV } = {}) { +function setup({ gh = fakeGitHub(), ec2 = fakeEc2(), ecs = fakeEcs(), env = ENV, now = () => T0 } = {}) { const logs = []; const sent = []; const sqs = { sendMessage: async (input) => { sent.push(input); return {}; } }; const handler = createHandler({ secret: { get: async () => ({ app_id: 42, private_key: PEM, webhook_secret: 'w' }) }, - fetch: gh.fetch, ec2, ecs, sqs, now: () => T0, env, log: (l) => logs.push(l), + fetch: gh.fetch, ec2, ecs, sqs, now, env, log: (l) => logs.push(l), }); return { handler, gh, ec2, ecs, sqs: sent, logs }; } @@ -313,6 +313,9 @@ const RECHECK = 300; const checkLater = (m) => ({ QueueUrl: ENV.QUEUE_URL, MessageBody: JSON.stringify(m), DelaySeconds: RECHECK }); const fleets = (ec2) => ec2.calls.filter((c) => c[0] === 'createFleet').length; const again = (sqs, i) => JSON.parse(sqs[i].MessageBody); +// A message after a launch: the count and the time of the job's last launch. +const stamped = (m, launches = 1, at = T0) => ({ ...m, launches, launched_at: at }); +const MIN = 60e3; test('the budget counts the stack\'s live instances by tag', async () => { const { handler, ec2 } = setup(); @@ -330,7 +333,7 @@ test('under the Spot budget, mixed sizes of both architectures leave room: the j const { handler, sqs } = setup({ ec2 }); await handler(sqsEvent({ ...MSG, vcpus: 8 })); assert.ok(launched(ec2)); - assert.deepEqual(sqs, [checkLater({ ...MSG, vcpus: 8 })]); + assert.deepEqual(sqs, [checkLater(stamped({ ...MSG, vcpus: 8 }))]); }); test('a job that brings its capacity exactly to the budget launches', async () => { @@ -339,7 +342,7 @@ test('a job that brings its capacity exactly to the budget launches', async () = const { handler, sqs } = setup({ ec2 }); await handler(sqsEvent(MSG)); assert.ok(launched(ec2)); - assert.deepEqual(sqs, [checkLater(MSG)]); + assert.deepEqual(sqs, [checkLater(stamped(MSG))]); }); test('a job that would go over the Spot budget goes back on the queue with a delay, logged in vCPUs', async () => { @@ -586,33 +589,38 @@ test('a failed RunTask removes the registration and fails the message, as a fail // checks again later, with its own message: a live runner tagged with the job // is served and the check repeats; none means a launch. -test('a launch schedules its own check: the same message, five minutes on', async () => { +test('a launch schedules its own check: its message, with the launch counted, five minutes on', async () => { const { handler, sqs } = setup(); await handler(sqsEvent(MSG)); - assert.deepEqual(sqs, [checkLater(MSG)]); + assert.deepEqual(sqs, [checkLater(stamped(MSG))]); }); test('a queued job whose launch produced no runner gets a new launch at its check', async () => { + let t = T0; const ec2 = fakeEc2(); // the launch left no instance behind - const { handler, sqs } = setup({ ec2 }); + const { handler, sqs } = setup({ ec2, now: () => t }); await handler(sqsEvent(MSG)); assert.equal(fleets(ec2), 1); + t = T0 + 15 * MIN; // a runner's idle exit later assert.deepEqual(await handler(sqsEvent(again(sqs, 0))), { batchItemFailures: [] }); assert.equal(fleets(ec2), 2, 'launched again'); - assert.deepEqual(sqs[1], checkLater(MSG), 'and checks again'); + assert.deepEqual(sqs[1], checkLater(stamped(MSG, 2, t)), 'and checks again'); }); test('a job held behind a concurrency group keeps its runner while it is up, and is launched again once that runner has gone', async () => { + let t = T0; const running = []; const ec2 = fakeEc2({ running }); - const { handler, sqs, gh } = setup({ ec2 }); + const { handler, sqs, gh } = setup({ ec2, now: () => t }); await handler(sqsEvent(MSG)); // the job is queued behind the held group assert.equal(fleets(ec2), 1); running.push({ job: MSG.job_id, capacity: 'spot', vcpus: 4, arch: 'x64', lifecycle: 'spot' }); // idle, group still held + t = T0 + 5 * MIN; await handler(sqsEvent(again(sqs, 0))); assert.equal(fleets(ec2), 1, 'a live runner for the job is not replaced'); - assert.deepEqual(sqs[1], checkLater(MSG), 'the check goes on'); + assert.deepEqual(sqs[1], checkLater(stamped(MSG)), 'the check goes on'); running[0].state = 'terminated'; // idled out before the group freed + t = T0 + 20 * MIN; await handler(sqsEvent(again(sqs, 1))); assert.equal(fleets(ec2), 2, 'with no runner left, the job is launched again'); // The group frees: the job is taken by that runner, so it is in progress, @@ -651,12 +659,14 @@ test('a check of a message from before the size labels launches the default shap const old = { ...MSG }; delete old.vcpus; delete old.arch; + let t = T0; const ec2 = fakeEc2(); - const { handler, sqs } = setup({ ec2 }); - await handler(sqsEvent(old)); + const { handler, sqs } = setup({ ec2, now: () => t }); + await handler(sqsEvent(old)); // no launch count: it launches at once + t = T0 + 15 * MIN; await handler(sqsEvent(again(sqs, 0))); assert.equal(fleets(ec2), 2); - assert.deepEqual(sqs[1], checkLater(old)); + assert.deepEqual(sqs[1], checkLater(stamped(old, 2, t))); }); test('a Fargate job served by its own task is not launched again, and its check goes on', async () => { @@ -666,3 +676,31 @@ test('a Fargate job served by its own task is not launched again, and its check assert.equal(runTaskInput(ecs), undefined, 'no RunTask'); assert.deepEqual(sqs, [checkLater(FG_MSG)]); }); + +// --- relaunches back off (#51, widened) ------------------------------------ +// A job whose runners never take it (the repo left the group, labels nothing +// matches) is relaunched no sooner than 15 minutes after its last launch, +// doubling for each launch after that, up to 60 minutes. + +test('a job whose runners never take it is relaunched with a growing gap: 15, then 30, then 60 minutes at most', async () => { + let t = T0; + const ec2 = fakeEc2(); // no runner ever stays up + const { handler, sqs } = setup({ ec2, now: () => t }); + const checkAt = async (minutes) => { t = T0 + minutes * 60e3; await handler(sqsEvent(again(sqs, sqs.length - 1))); }; + await handler(sqsEvent(MSG)); + assert.equal(fleets(ec2), 1, 'the first launch, at 0'); + await checkAt(5); + assert.equal(fleets(ec2), 1, 'too soon at 5'); + await checkAt(15); + assert.equal(fleets(ec2), 2, 'the second launch, 15 minutes on'); + await checkAt(40); + assert.equal(fleets(ec2), 2, 'too soon, 25 minutes after the second'); + await checkAt(45); + assert.equal(fleets(ec2), 3, 'the third launch, 30 minutes after the second'); + await checkAt(100); + assert.equal(fleets(ec2), 3, 'too soon, 55 minutes after the third'); + await checkAt(105); + assert.equal(fleets(ec2), 4, 'the fourth launch, 60 minutes after the third'); + await checkAt(165); + assert.equal(fleets(ec2), 5, 'the gap stays at 60 minutes'); +}); From 351e4012a868c97b03ebf8c02610d90f95cdb741 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patrick=20Sodr=C3=A9?= Date: Fri, 20 Nov 2026 00:56:26 -0500 Subject: [PATCH 4/4] Launcher tests: a check too soon to relaunch still sends the next check The backoff test replayed the last message whether or not a new one was sent, so a waiting check that dropped the job's message would have passed. Each check in it now must send exactly one next check, and the one too soon keeps the launch count and time. Co-Authored-By: Claude Opus 5.5 --- test/launcher.test.js | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/test/launcher.test.js b/test/launcher.test.js index 5e134c6..a1a5ce7 100644 --- a/test/launcher.test.js +++ b/test/launcher.test.js @@ -686,11 +686,18 @@ test('a job whose runners never take it is relaunched with a growing gap: 15, th let t = T0; const ec2 = fakeEc2(); // no runner ever stays up const { handler, sqs } = setup({ ec2, now: () => t }); - const checkAt = async (minutes) => { t = T0 + minutes * 60e3; await handler(sqsEvent(again(sqs, sqs.length - 1))); }; + // Every check, too soon or not, must send the next one, or the job is lost. + const checkAt = async (minutes) => { + t = T0 + minutes * 60e3; + const before = sqs.length; + await handler(sqsEvent(again(sqs, sqs.length - 1))); + assert.equal(sqs.length, before + 1, `the check at ${minutes} sends the next check`); + }; await handler(sqsEvent(MSG)); assert.equal(fleets(ec2), 1, 'the first launch, at 0'); await checkAt(5); assert.equal(fleets(ec2), 1, 'too soon at 5'); + assert.deepEqual(sqs.at(-1), checkLater(stamped(MSG)), 'a check too soon keeps the launch count and time'); await checkAt(15); assert.equal(fleets(ec2), 2, 'the second launch, 15 minutes on'); await checkAt(40);