diff --git a/docs/stack.md b/docs/stack.md index 9a41b5c..ad23fcc 100644 --- a/docs/stack.md +++ b/docs/stack.md @@ -318,8 +318,27 @@ failures per message), for each job: 7. If no instance or task launches, it removes the registration it just minted and fails the message, which comes back 120 s later. -There is no de-duplication on job id. A duplicate delivery launches a second -runner at worst, and the idle exit and the sweeper remove it. +**The check that follows (#51).** Every launch, and every check that finds the +job still queued and served, sends the job's own message again, delayed 300 s +(`RECHECK_SECONDS`), once the launch is done. A check first looks at the +machines this stack tags with the job (`ci-runner:job-id`): a `pending` or +`running` instance, or a task not yet told to stop. If one is serving the job, +the check sends its own message on and launches nothing (its log line says +`served`). If none is, the job is launched as above, but no sooner than 15 +minutes after its last launch, a gap that doubles for each launch after that, up +to 60 minutes (its log line says `waiting` until then). So a job whose runners +never take it (the repo left the group, labels nothing matches) costs about one +launch an hour, not one every check. The message carries `launches` and +`launched_at` for this; one without them (the webhook's, a swap's re-queue) has +had no launch and launches at once. The checks end when GitHub +no longer has the job queued, which drops the message. This is why a job held +behind a concurrency group the first job holds gets a runner once the group +frees: a runner that idled out first is replaced at the next check. The sweeper +does not do this, since it has no list of queued jobs. Not yet checked live: +the behaviour is covered with fakes in `test/launcher.test.js`. + +There is no de-duplication on job id beyond that check. A duplicate delivery +launches a second runner at worst, and the idle exit and the sweeper remove it. `cloudformation deploy` keeps a parameter's previous value unless it's overridden, so after lowering a budget for a test, restore it explicitly diff --git a/docs/traps.md b/docs/traps.md index 64a78f1..4bfde87 100644 --- a/docs/traps.md +++ b/docs/traps.md @@ -9,7 +9,7 @@ Symptoms seen on the stack, and their causes. A quoted section name is in | `permission denied` on `/var/run/docker.sock` | the host's `docker` group isn't gid 123 | | `setup-python` fails, or picks up a broken Python | `AGENT_TOOLSDIRECTORY` isn't `/opt/hostedtoolcache` | | A deploy waits 60 minutes, then fails on the state lock | an `apply` ran on Spot and was reclaimed; it must be `ci-ondemand` | -| A job sits "Waiting for a runner" | its capacity's vCPU budget is full, the repo isn't in the group, the App isn't installed on it, or the repo is public. Also check the label contract: `self-hosted`, exactly one of `ci-spot`/`ci-ondemand`/`ci-fargate-spot`/`ci-fargate`, exactly one size label (`ci-cpu-2`/`-4`/`-8`/`-16`), exactly one of `x64`/`arm64`, and otherwise only `linux`. The launcher's log tells the first case (`deferred`) apart from the rest (no line at all). A job that queued before the repo was turned on stays waiting after the fix: cancel and re-run it | +| A job sits "Waiting for a runner" | its capacity's vCPU budget is full, the repo isn't in the group, the App isn't installed on it, or the repo is public. Also check the label contract: `self-hosted`, exactly one of `ci-spot`/`ci-ondemand`/`ci-fargate-spot`/`ci-fargate`, exactly one size label (`ci-cpu-2`/`-4`/`-8`/`-16`), exactly one of `x64`/`arm64`, and otherwise only `linux`. The launcher's log tells the first case (`deferred`) apart from the rest (no line at all). A job held behind a concurrency group shows `served` while its runner is up; once that runner idles out, a later check (5 minutes apart, see "Launching for queued jobs" in `stack.md`) launches another, no sooner than 15 minutes after the last launch and backing off to an hour (`waiting` in the log). A job that queued before the repo was turned on stays waiting after the fix: cancel and re-run it | | A job that ran on our runners before now sits "Waiting for a runner", and the webhook's log shows no `enqueued` line for it | missing size or architecture label: its `runs-on` names only its capacity (`self-hosted`, `ci-spot`). There are no defaults; add a size and an architecture label (`ci-cpu-4` and `x64` keep the old shape), then cancel and re-run the job | | An `arm64` runner's instance comes up and terminates without taking its job, and its console shows `no matching manifest for linux/arm64` | `ghcr.io/zeroae/ci-runner:latest` has no arm64 entry yet: the image workflow hasn't pushed a multi-arch `:latest` since it began building arm64. Run the image workflow on `main`, then re-run the job | | A job's runner was launched for another job (`$RUNNER_NAME` ends in another job id), and that other job sits "Waiting for a runner" | a swap. The webhook repairs it on the job's `in_progress` (its log says `swap`), and the launcher then launches a runner for the other job. No `swap` line: check the webhook's log for errors (its role needs `ec2:DescribeInstances` and `ec2:CreateTags`, and for a task `ecs:ListTasks`, `ecs:DescribeTasks` and `ecs:TagResource`); cancel and re-run the waiting job | diff --git a/functions/launcher.js b/functions/launcher.js index 47f3a2b..3971c8c 100644 --- a/functions/launcher.js +++ b/functions/launcher.js @@ -26,6 +26,18 @@ const ecsTag = ([key, v]) => ({ key, value: String(v).replace(ECS_TAG_CHARS, '_' // without the ci-runner: prefix. The webhook and the notifier carry copies. const tagsOf = (tags) => Object.fromEntries((tags || []).map((x) => [(x.Key ?? x.key).replace(/^ci-runner:/, ''), x.Value ?? x.value])); +// A queued job is checked again this many seconds after each check (#51): a +// runner that idled out, or never got the job, is replaced by the next check. +// SQS allows a delay of up to 900 seconds. +const RECHECK_SECONDS = 300; +// A job whose runners never take it (the repo left the group, labels nothing +// matches) is relaunched no sooner than this after its last launch: 15 minutes, +// a runner's idle exit, for the first relaunch, doubling for each after, up to +// 60 minutes. A held concurrency group still frees into a runner within the +// hour. The message carries the count and the time of the job's last launch; +// one without them (the webhook's, a swap's re-queue) has had none. +const relaunchGapMs = (launches) => Math.min(15 * 2 ** (launches - 1), 60) * 60e3; + const b64u = (s) => Buffer.from(s).toString('base64url'); function appJwt(appId, pem, nowMs) { @@ -50,6 +62,8 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l const org = env.GITHUB_ORG; const subnets = env.SUBNET_IDS.split(','); const budgets = { spot: env.SPOT_VCPU_CAP, 'on-demand': env.ONDEMAND_VCPU_CAP, 'fargate-spot': env.FARGATE_SPOT_VCPU_CAP, fargate: env.FARGATE_VCPU_CAP }; + // The job's own message, delayed: its next check (#51). + const checkLater = (m) => ({ QueueUrl: env.QUEUE_URL, MessageBody: JSON.stringify(m), DelaySeconds: RECHECK_SECONDS }); const tokens = new Map(); // installation id -> { token, until } let app; let groupId; @@ -87,12 +101,13 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l } } - // vCPUs of this stack's live instances on one capacity, from the shape tags. - // An instance launched before them has no tags: it has the default shape's - // vCPUs, and its capacity is Spot only with the Spot marker (On-Demand has - // no InstanceLifecycle). - async function vcpusIn(type) { - let n = 0; + // This stack's instances that are not terminated, one record each: its job + // (the ci-runner:job-id tag), its shape from the shape tags, and whether it + // is still serving that job (pending or running). An instance launched before + // the shape tags has the default shape, and its capacity is Spot only with + // the Spot marker (On-Demand has no InstanceLifecycle). + async function instancesLive() { + const out = []; let NextToken; do { const r = await ec2.describeInstances({ @@ -106,12 +121,12 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l for (const i of res.Instances || []) { const t = tagsOf(i.Tags); const capacity = CAPACITY[t.capacity] || (i.InstanceLifecycle === 'spot' ? 'spot' : 'on-demand'); - if (capacity === type) n += Number(t.vcpus) || DEFAULT_SHAPE.vcpus; + out.push({ id: i.InstanceId, job: t['job-id'], capacity, vcpus: Number(t.vcpus) || DEFAULT_SHAPE.vcpus, serving: ['pending', 'running'].includes(i.State.Name) }); } } NextToken = r.NextToken; } while (NextToken); - return n; + return out; } // The tags a runner's instance or task carries: the job, for the swap @@ -162,11 +177,11 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l return arn; } - // vCPUs of the cluster's tasks on one Fargate capacity, from the shape - // tags. A task counts until it is STOPPED: one told to stop (desired - // STOPPED) still holds its vCPUs while it deprovisions. - async function taskVcpusIn(type) { - let n = 0; + // The cluster's tasks, one record each, as instancesLive. A task holds its + // vCPUs until it is STOPPED: one told to stop (desired STOPPED) still holds + // them while it deprovisions, but is no longer serving its job. + async function tasksLive() { + const out = []; for (const desiredStatus of ['RUNNING', 'STOPPED']) { let nextToken; do { @@ -175,13 +190,14 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l const d = await ecs.describeTasks({ cluster: env.CLUSTER, tasks: r.taskArns, include: ['TAGS'] }); for (const task of d.tasks || []) { const t = tagsOf(task.tags); - if (task.lastStatus !== 'STOPPED' && CAPACITY[t.capacity] === type) n += Number(t.vcpus) || DEFAULT_SHAPE.vcpus; + if (task.lastStatus === 'STOPPED') continue; + out.push({ id: task.taskArn, job: t['job-id'], capacity: CAPACITY[t.capacity], vcpus: Number(t.vcpus) || DEFAULT_SHAPE.vcpus, serving: desiredStatus === 'RUNNING' }); } } nextToken = r.nextToken; } while (nextToken); } - return n; + return out; } async function launchInstance(m, name, runnerId, jitConfig) { @@ -229,12 +245,29 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l if (!j) return log(JSON.stringify({ ...job, dropped: 'job not found' })); if (j.status !== 'queued') return log(JSON.stringify({ ...job, dropped: j.status })); + // One listing serves both checks. A job whose runner is still serving it is + // not launched again: its check goes on, so a runner that idles out without + // the job is replaced by a later check (#51). + const type = CAPACITY[m.label]; + const machines = await (PROVIDER[m.label] ? tasksLive() : instancesLive()); + const served = machines.find((x) => x.serving && x.job === String(m.job_id)); + if (served) { + await sqs.sendMessage(checkLater(m)); + return log(JSON.stringify({ ...job, served: served.id })); + } + + // Launched before and not served now: wait out the gap before a relaunch. + const launches = Number.isInteger(m.launches) && m.launches > 0 ? m.launches : 0; + if (launches && now() - m.launched_at < relaunchGapMs(launches)) { + await sqs.sendMessage(checkLater(m)); + return log(JSON.stringify({ ...job, waiting: `relaunch ${launches + 1} from ${new Date(m.launched_at + relaunchGapMs(launches)).toISOString()}` })); + } + // A soft budget: launches checked at the same moment may overshoot it. A // job that would go over it goes back on the queue as a new message, so // retries don't pile up towards the dead-letter queue. - const type = CAPACITY[m.label]; const budget = Number(budgets[type]); - const held = await (PROVIDER[m.label] ? taskVcpusIn(type) : vcpusIn(type)); + const held = machines.filter((x) => x.capacity === type).reduce((n, x) => n + x.vcpus, 0); const { vcpus } = shapeOf(m); if (held + vcpus > budget) { await sqs.sendMessage({ QueueUrl: env.QUEUE_URL, MessageBody: JSON.stringify(m), DelaySeconds: 60 }); @@ -258,6 +291,9 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l await gh('DELETE', `/orgs/${org}/actions/runners/${r.runner.id}`, token).catch(() => {}); throw e; } + // After the launch, so a failed send here never removes a live runner: the + // retry then finds it served and sends the check. + await sqs.sendMessage(checkLater({ ...m, launches: launches + 1, launched_at: now() })); } return async (event) => { diff --git a/test/launcher-ecs.test.js b/test/launcher-ecs.test.js index 2b7d290..418d12d 100644 --- a/test/launcher-ecs.test.js +++ b/test/launcher-ecs.test.js @@ -73,12 +73,15 @@ const job = (label, vcpus, arch, extra = {}) => ({ Records: [{ messageId: 'm0', }) }] }); test('ECS accepts the launcher\'s RunTask for every Fargate capacity, size and architecture', { skip }, async () => { + // One job id per launch: a task still serving a job is not launched again (#51), + // and the emulator keeps the tasks of earlier launches. + let jobId = 1000; for (const [label, provider] of [['ci-fargate-spot', 'FARGATE_SPOT'], ['ci-fargate', 'FARGATE']]) { for (const vcpus of [2, 4, 8, 16]) { for (const arch of ['x64', 'arm64']) { const what = `${label} ${vcpus} ${arch}`; const { handler, logs } = launcher(fakeFetch()); - assert.deepEqual(await handler(job(label, vcpus, arch)), { batchItemFailures: [] }, `${what}: ${logs.join('\n')}`); + assert.deepEqual(await handler(job(label, vcpus, arch, { job_id: jobId++ })), { batchItemFailures: [] }, `${what}: ${logs.join('\n')}`); const arn = JSON.parse(logs.find((l) => l.includes('"launched"'))).launched; const [task] = (await ecs.describeTasks({ cluster: 'ci-runner', tasks: [arn], include: ['TAGS'] })).tasks; assert.equal(task.capacityProviderName, provider, what); diff --git a/test/launcher.test.js b/test/launcher.test.js index 8dfbb4c..a1a5ce7 100644 --- a/test/launcher.test.js +++ b/test/launcher.test.js @@ -33,6 +33,7 @@ function verifyJwt(auth) { // A fake GitHub: routes keyed "METHOD path". Records every call. function fakeGitHub({ jobStatus = 'queued', jobHttp = 200, jit = { status: 201 }, groups = [{ id: 1, name: 'Default' }, { id: 3, name: 'ci-runner' }] } = {}) { const calls = []; + const job = { status: jobStatus }; // a test can change it, as GitHub does const fetch = async (url, init = {}) => { const u = new URL(url); const key = `${init.method || 'GET'} ${u.pathname}`; @@ -45,7 +46,7 @@ function fakeGitHub({ jobStatus = 'queued', jobHttp = 200, jit = { status: 201 } return res(201, { token: 'ghs_install', expires_at: new Date(T0 + 3600e3).toISOString() }); } assert.equal(init.headers.authorization, 'Bearer ghs_install', key); - if (key === 'GET /repos/zeroae/app/actions/jobs/555') return res(jobHttp, jobHttp === 200 ? { id: 555, status: jobStatus } : { message: 'x' }); + if (key === 'GET /repos/zeroae/app/actions/jobs/555') return res(jobHttp, jobHttp === 200 ? { id: 555, status: job.status } : { message: 'x' }); if (key === 'GET /orgs/zeroae/actions/runner-groups') return res(200, { total_count: groups.length, runner_groups: groups }); if (key === 'POST /orgs/zeroae/actions/runners/generate-jitconfig') { return jit.status === 201 ? res(201, { runner: { id: 77, name: body.name }, encoded_jit_config: JIT }) : res(jit.status, { message: 'nope' }); @@ -53,7 +54,7 @@ function fakeGitHub({ jobStatus = 'queued', jobHttp = 200, jit = { status: 201 } if (key === 'DELETE /orgs/zeroae/actions/runners/77') return res(204, null); throw new Error('unexpected ' + key); }; - return { fetch, calls, keys: () => calls.map((c) => c.key) }; + return { fetch, calls, job, keys: () => calls.map((c) => c.key) }; } // `running` lists the stack's instances: { lifecycle: 'spot' | undefined, state, @@ -62,10 +63,11 @@ function fakeEc2({ fleet = { Instances: [{ InstanceIds: ['i-0123'] }], Errors: [ const calls = []; return { calls, + running, describeInstances: async (input) => { calls.push(['describe', input]); const all = running.map((r, i) => ({ InstanceId: `i-r${i}`, State: { Name: r.state || 'running' }, ...(r.lifecycle ? { InstanceLifecycle: r.lifecycle } : {}), - Tags: [['ci-runner:stack', 'ci-runner'], ['ci-runner:capacity', r.capacity], ['ci-runner:vcpus', r.vcpus], ['ci-runner:arch', r.arch]] + Tags: [['ci-runner:stack', 'ci-runner'], ['ci-runner:capacity', r.capacity], ['ci-runner:vcpus', r.vcpus], ['ci-runner:arch', r.arch], ['ci-runner:job-id', r.job]] .filter(([, v]) => v !== undefined).map(([Key, Value]) => ({ Key, Value: String(Value) })) })); // Two pages, to check NextToken is followed. if (!input.NextToken) return { Reservations: [{ Instances: all.slice(0, 1) }], NextToken: all.length > 1 ? 'p2' : undefined }; @@ -84,7 +86,7 @@ function fakeEcs({ tasks = [], run } = {}) { const calls = []; const all = tasks.map((t, i) => ({ taskArn: `arn:aws:ecs:us-east-1:1:task/ci-runner/t${i}`, desiredStatus: t.desired || 'RUNNING', lastStatus: t.last || 'RUNNING', - tags: [['ci-runner:stack', 'ci-runner'], ['ci-runner:capacity', t.capacity], ['ci-runner:vcpus', t.vcpus]] + tags: [['ci-runner:stack', 'ci-runner'], ['ci-runner:capacity', t.capacity], ['ci-runner:vcpus', t.vcpus], ['ci-runner:job-id', t.job]] .filter(([, v]) => v !== undefined).map(([key, value]) => ({ key, value: String(value) })), })); return { @@ -110,13 +112,13 @@ function fakeEcs({ tasks = [], run } = {}) { }; } -function setup({ gh = fakeGitHub(), ec2 = fakeEc2(), ecs = fakeEcs(), env = ENV } = {}) { +function setup({ gh = fakeGitHub(), ec2 = fakeEc2(), ecs = fakeEcs(), env = ENV, now = () => T0 } = {}) { const logs = []; const sent = []; const sqs = { sendMessage: async (input) => { sent.push(input); return {}; } }; const handler = createHandler({ secret: { get: async () => ({ app_id: 42, private_key: PEM, webhook_secret: 'w' }) }, - fetch: gh.fetch, ec2, ecs, sqs, now: () => T0, env, log: (l) => logs.push(l), + fetch: gh.fetch, ec2, ecs, sqs, now, env, log: (l) => logs.push(l), }); return { handler, gh, ec2, ecs, sqs: sent, logs }; } @@ -306,6 +308,14 @@ const OLD_SPOT = { lifecycle: 'spot' }; const OLD_OD = {}; const OD_MSG = { ...MSG, label: 'ci-ondemand', labels: ['self-hosted', 'ci-ondemand'] }; const launched = (ec2) => ec2.calls.some((c) => c[0] === 'createFleet'); +// The check a queued job schedules for itself (#51): the same message, later. +const RECHECK = 300; +const checkLater = (m) => ({ QueueUrl: ENV.QUEUE_URL, MessageBody: JSON.stringify(m), DelaySeconds: RECHECK }); +const fleets = (ec2) => ec2.calls.filter((c) => c[0] === 'createFleet').length; +const again = (sqs, i) => JSON.parse(sqs[i].MessageBody); +// A message after a launch: the count and the time of the job's last launch. +const stamped = (m, launches = 1, at = T0) => ({ ...m, launches, launched_at: at }); +const MIN = 60e3; test('the budget counts the stack\'s live instances by tag', async () => { const { handler, ec2 } = setup(); @@ -323,7 +333,7 @@ test('under the Spot budget, mixed sizes of both architectures leave room: the j const { handler, sqs } = setup({ ec2 }); await handler(sqsEvent({ ...MSG, vcpus: 8 })); assert.ok(launched(ec2)); - assert.deepEqual(sqs, []); + assert.deepEqual(sqs, [checkLater(stamped({ ...MSG, vcpus: 8 }))]); }); test('a job that brings its capacity exactly to the budget launches', async () => { @@ -332,7 +342,7 @@ test('a job that brings its capacity exactly to the budget launches', async () = const { handler, sqs } = setup({ ec2 }); await handler(sqsEvent(MSG)); assert.ok(launched(ec2)); - assert.deepEqual(sqs, []); + assert.deepEqual(sqs, [checkLater(stamped(MSG))]); }); test('a job that would go over the Spot budget goes back on the queue with a delay, logged in vCPUs', async () => { @@ -572,3 +582,132 @@ test('a failed RunTask removes the registration and fails the message, as a fail assert.match(err, run === failed ? /^no task launched: Capacity is unavailable at this time: FARGATE_SPOT$/ : /AccessDenied/); } }); + +// --- a queued job keeps its check (#51) ------------------------------------ +// A launch's runner can go without taking the job (it idles out, or the job +// waits behind a concurrency group the first job holds). So each queued job +// checks again later, with its own message: a live runner tagged with the job +// is served and the check repeats; none means a launch. + +test('a launch schedules its own check: its message, with the launch counted, five minutes on', async () => { + const { handler, sqs } = setup(); + await handler(sqsEvent(MSG)); + assert.deepEqual(sqs, [checkLater(stamped(MSG))]); +}); + +test('a queued job whose launch produced no runner gets a new launch at its check', async () => { + let t = T0; + const ec2 = fakeEc2(); // the launch left no instance behind + const { handler, sqs } = setup({ ec2, now: () => t }); + await handler(sqsEvent(MSG)); + assert.equal(fleets(ec2), 1); + t = T0 + 15 * MIN; // a runner's idle exit later + assert.deepEqual(await handler(sqsEvent(again(sqs, 0))), { batchItemFailures: [] }); + assert.equal(fleets(ec2), 2, 'launched again'); + assert.deepEqual(sqs[1], checkLater(stamped(MSG, 2, t)), 'and checks again'); +}); + +test('a job held behind a concurrency group keeps its runner while it is up, and is launched again once that runner has gone', async () => { + let t = T0; + const running = []; + const ec2 = fakeEc2({ running }); + const { handler, sqs, gh } = setup({ ec2, now: () => t }); + await handler(sqsEvent(MSG)); // the job is queued behind the held group + assert.equal(fleets(ec2), 1); + running.push({ job: MSG.job_id, capacity: 'spot', vcpus: 4, arch: 'x64', lifecycle: 'spot' }); // idle, group still held + t = T0 + 5 * MIN; + await handler(sqsEvent(again(sqs, 0))); + assert.equal(fleets(ec2), 1, 'a live runner for the job is not replaced'); + assert.deepEqual(sqs[1], checkLater(stamped(MSG)), 'the check goes on'); + running[0].state = 'terminated'; // idled out before the group freed + t = T0 + 20 * MIN; + await handler(sqsEvent(again(sqs, 1))); + assert.equal(fleets(ec2), 2, 'with no runner left, the job is launched again'); + // The group frees: the job is taken by that runner, so it is in progress, + // and the next check drops the job and schedules nothing. + gh.job.status = 'in_progress'; + await handler(sqsEvent(again(sqs, 2))); + assert.equal(fleets(ec2), 2, 'not launched again'); + assert.equal(sqs.length, 3, 'and no further check'); +}); + +// GitHub reports a cancelled job as status "completed" with conclusion +// "cancelled" (probed on zeroae/ci-runner, run 43221273888, 2026-11-20), so +// "completed" covers the cancelled case too. +test('a check for a job that has started, finished or was cancelled launches nothing and schedules no further check', async () => { + for (const jobStatus of ['in_progress', 'completed']) { + const ec2 = fakeEc2({ running: [{ job: MSG.job_id, capacity: 'spot', vcpus: 4, arch: 'x64', lifecycle: 'spot' }] }); + const { handler, sqs } = setup({ gh: fakeGitHub({ jobStatus }), ec2 }); + assert.deepEqual(await handler(sqsEvent(MSG)), { batchItemFailures: [] }); + assert.equal(fleets(ec2), 0, jobStatus); + assert.deepEqual(sqs, [], jobStatus); + } +}); + +test('two checks in a row launch one runner, not two', async () => { + const running = []; + const ec2 = fakeEc2({ running }); + const { handler, sqs } = setup({ ec2 }); + await handler(sqsEvent(MSG)); + running.push({ job: MSG.job_id, capacity: 'spot', vcpus: 4, arch: 'x64', lifecycle: 'spot' }); // its runner is up + await handler(sqsEvent(MSG)); // a repeat delivery of the original + await handler(sqsEvent(again(sqs, 0))); + assert.equal(fleets(ec2), 1); +}); + +test('a check of a message from before the size labels launches the default shape, and checks again', async () => { + const old = { ...MSG }; + delete old.vcpus; + delete old.arch; + let t = T0; + const ec2 = fakeEc2(); + const { handler, sqs } = setup({ ec2, now: () => t }); + await handler(sqsEvent(old)); // no launch count: it launches at once + t = T0 + 15 * MIN; + await handler(sqsEvent(again(sqs, 0))); + assert.equal(fleets(ec2), 2); + assert.deepEqual(sqs[1], checkLater(stamped(old, 2, t))); +}); + +test('a Fargate job served by its own task is not launched again, and its check goes on', async () => { + const ecs = fakeEcs({ tasks: [{ capacity: 'fargate-spot', vcpus: 4, job: FG_MSG.job_id }] }); + const { handler, sqs } = setup({ ecs }); + assert.deepEqual(await handler(sqsEvent(FG_MSG)), { batchItemFailures: [] }); + assert.equal(runTaskInput(ecs), undefined, 'no RunTask'); + assert.deepEqual(sqs, [checkLater(FG_MSG)]); +}); + +// --- relaunches back off (#51, widened) ------------------------------------ +// A job whose runners never take it (the repo left the group, labels nothing +// matches) is relaunched no sooner than 15 minutes after its last launch, +// doubling for each launch after that, up to 60 minutes. + +test('a job whose runners never take it is relaunched with a growing gap: 15, then 30, then 60 minutes at most', async () => { + let t = T0; + const ec2 = fakeEc2(); // no runner ever stays up + const { handler, sqs } = setup({ ec2, now: () => t }); + // Every check, too soon or not, must send the next one, or the job is lost. + const checkAt = async (minutes) => { + t = T0 + minutes * 60e3; + const before = sqs.length; + await handler(sqsEvent(again(sqs, sqs.length - 1))); + assert.equal(sqs.length, before + 1, `the check at ${minutes} sends the next check`); + }; + await handler(sqsEvent(MSG)); + assert.equal(fleets(ec2), 1, 'the first launch, at 0'); + await checkAt(5); + assert.equal(fleets(ec2), 1, 'too soon at 5'); + assert.deepEqual(sqs.at(-1), checkLater(stamped(MSG)), 'a check too soon keeps the launch count and time'); + await checkAt(15); + assert.equal(fleets(ec2), 2, 'the second launch, 15 minutes on'); + await checkAt(40); + assert.equal(fleets(ec2), 2, 'too soon, 25 minutes after the second'); + await checkAt(45); + assert.equal(fleets(ec2), 3, 'the third launch, 30 minutes after the second'); + await checkAt(100); + assert.equal(fleets(ec2), 3, 'too soon, 55 minutes after the third'); + await checkAt(105); + assert.equal(fleets(ec2), 4, 'the fourth launch, 60 minutes after the third'); + await checkAt(165); + assert.equal(fleets(ec2), 5, 'the gap stays at 60 minutes'); +}); diff --git a/test/tasks-ecs.test.js b/test/tasks-ecs.test.js index 679965d..11a7395 100644 --- a/test/tasks-ecs.test.js +++ b/test/tasks-ecs.test.js @@ -118,7 +118,7 @@ function fakeGitHub(runners = [], job = {}) { async function launch() { const logs = []; const h = launcher.createHandler({ - secret, fetch: fakeGitHub().fetch, ecs, ec2: {}, sqs: {}, now: Date.now, env, log: (l) => logs.push(l), + secret, fetch: fakeGitHub().fetch, ecs, ec2: {}, sqs: { sendMessage: async () => ({}) }, now: Date.now, env, log: (l) => logs.push(l), }); const body = JSON.stringify({ repo: 'zeroae/lib', job_id: 999, run_id: 888, head_sha: 'fff000', job_name: JOB_NAMES[999],