Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 21 additions & 2 deletions docs/stack.md
Original file line number Diff line number Diff line change
Expand Up @@ -318,8 +318,27 @@ failures per message), for each job:
7. If no instance or task launches, it removes the registration it just
minted and fails the message, which comes back 120 s later.

There is no de-duplication on job id. A duplicate delivery launches a second
runner at worst, and the idle exit and the sweeper remove it.
**The check that follows (#51).** Every launch, and every check that finds the
job still queued and served, sends the job's own message again, delayed 300 s
(`RECHECK_SECONDS`), once the launch is done. A check first looks at the
machines this stack tags with the job (`ci-runner:job-id`): a `pending` or
`running` instance, or a task not yet told to stop. If one is serving the job,
the check sends its own message on and launches nothing (its log line says
`served`). If none is, the job is launched as above, but no sooner than 15
minutes after its last launch, a gap that doubles for each launch after that, up
to 60 minutes (its log line says `waiting` until then). So a job whose runners
never take it (the repo left the group, labels nothing matches) costs about one
launch an hour, not one every check. The message carries `launches` and
`launched_at` for this; one without them (the webhook's, a swap's re-queue) has
had no launch and launches at once. The checks end when GitHub
no longer has the job queued, which drops the message. This is why a job held
behind a concurrency group the first job holds gets a runner once the group
frees: a runner that idled out first is replaced at the next check. The sweeper
does not do this, since it has no list of queued jobs. Not yet checked live:
the behaviour is covered with fakes in `test/launcher.test.js`.

There is no de-duplication on job id beyond that check. A duplicate delivery
launches a second runner at worst, and the idle exit and the sweeper remove it.

`cloudformation deploy` keeps a parameter's previous value unless it's
overridden, so after lowering a budget for a test, restore it explicitly
Expand Down
2 changes: 1 addition & 1 deletion docs/traps.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Symptoms seen on the stack, and their causes. A quoted section name is in
| `permission denied` on `/var/run/docker.sock` | the host's `docker` group isn't gid 123 |
| `setup-python` fails, or picks up a broken Python | `AGENT_TOOLSDIRECTORY` isn't `/opt/hostedtoolcache` |
| A deploy waits 60 minutes, then fails on the state lock | an `apply` ran on Spot and was reclaimed; it must be `ci-ondemand` |
| A job sits "Waiting for a runner" | its capacity's vCPU budget is full, the repo isn't in the group, the App isn't installed on it, or the repo is public. Also check the label contract: `self-hosted`, exactly one of `ci-spot`/`ci-ondemand`/`ci-fargate-spot`/`ci-fargate`, exactly one size label (`ci-cpu-2`/`-4`/`-8`/`-16`), exactly one of `x64`/`arm64`, and otherwise only `linux`. The launcher's log tells the first case (`deferred`) apart from the rest (no line at all). A job that queued before the repo was turned on stays waiting after the fix: cancel and re-run it |
| A job sits "Waiting for a runner" | its capacity's vCPU budget is full, the repo isn't in the group, the App isn't installed on it, or the repo is public. Also check the label contract: `self-hosted`, exactly one of `ci-spot`/`ci-ondemand`/`ci-fargate-spot`/`ci-fargate`, exactly one size label (`ci-cpu-2`/`-4`/`-8`/`-16`), exactly one of `x64`/`arm64`, and otherwise only `linux`. The launcher's log tells the first case (`deferred`) apart from the rest (no line at all). A job held behind a concurrency group shows `served` while its runner is up; once that runner idles out, a later check (5 minutes apart, see "Launching for queued jobs" in `stack.md`) launches another, no sooner than 15 minutes after the last launch and backing off to an hour (`waiting` in the log). A job that queued before the repo was turned on stays waiting after the fix: cancel and re-run it |
| A job that ran on our runners before now sits "Waiting for a runner", and the webhook's log shows no `enqueued` line for it | missing size or architecture label: its `runs-on` names only its capacity (`self-hosted`, `ci-spot`). There are no defaults; add a size and an architecture label (`ci-cpu-4` and `x64` keep the old shape), then cancel and re-run the job |
| An `arm64` runner's instance comes up and terminates without taking its job, and its console shows `no matching manifest for linux/arm64` | `ghcr.io/zeroae/ci-runner:latest` has no arm64 entry yet: the image workflow hasn't pushed a multi-arch `:latest` since it began building arm64. Run the image workflow on `main`, then re-run the job |
| A job's runner was launched for another job (`$RUNNER_NAME` ends in another job id), and that other job sits "Waiting for a runner" | a swap. The webhook repairs it on the job's `in_progress` (its log says `swap`), and the launcher then launches a runner for the other job. No `swap` line: check the webhook's log for errors (its role needs `ec2:DescribeInstances` and `ec2:CreateTags`, and for a task `ecs:ListTasks`, `ecs:DescribeTasks` and `ecs:TagResource`); cancel and re-run the waiting job |
Expand Down
70 changes: 53 additions & 17 deletions functions/launcher.js
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,18 @@ const ecsTag = ([key, v]) => ({ key, value: String(v).replace(ECS_TAG_CHARS, '_'
// without the ci-runner: prefix. The webhook and the notifier carry copies.
const tagsOf = (tags) => Object.fromEntries((tags || []).map((x) => [(x.Key ?? x.key).replace(/^ci-runner:/, ''), x.Value ?? x.value]));

// A queued job is checked again this many seconds after each check (#51): a
// runner that idled out, or never got the job, is replaced by the next check.
// SQS allows a delay of up to 900 seconds.
const RECHECK_SECONDS = 300;
// A job whose runners never take it (the repo left the group, labels nothing
// matches) is relaunched no sooner than this after its last launch: 15 minutes,
// a runner's idle exit, for the first relaunch, doubling for each after, up to
// 60 minutes. A held concurrency group still frees into a runner within the
// hour. The message carries the count and the time of the job's last launch;
// one without them (the webhook's, a swap's re-queue) has had none.
const relaunchGapMs = (launches) => Math.min(15 * 2 ** (launches - 1), 60) * 60e3;

const b64u = (s) => Buffer.from(s).toString('base64url');

function appJwt(appId, pem, nowMs) {
Expand All @@ -50,6 +62,8 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l
const org = env.GITHUB_ORG;
const subnets = env.SUBNET_IDS.split(',');
const budgets = { spot: env.SPOT_VCPU_CAP, 'on-demand': env.ONDEMAND_VCPU_CAP, 'fargate-spot': env.FARGATE_SPOT_VCPU_CAP, fargate: env.FARGATE_VCPU_CAP };
// The job's own message, delayed: its next check (#51).
const checkLater = (m) => ({ QueueUrl: env.QUEUE_URL, MessageBody: JSON.stringify(m), DelaySeconds: RECHECK_SECONDS });
const tokens = new Map(); // installation id -> { token, until }
let app;
let groupId;
Expand Down Expand Up @@ -87,12 +101,13 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l
}
}

// vCPUs of this stack's live instances on one capacity, from the shape tags.
// An instance launched before them has no tags: it has the default shape's
// vCPUs, and its capacity is Spot only with the Spot marker (On-Demand has
// no InstanceLifecycle).
async function vcpusIn(type) {
let n = 0;
// This stack's instances that are not terminated, one record each: its job
// (the ci-runner:job-id tag), its shape from the shape tags, and whether it
// is still serving that job (pending or running). An instance launched before
// the shape tags has the default shape, and its capacity is Spot only with
// the Spot marker (On-Demand has no InstanceLifecycle).
async function instancesLive() {
const out = [];
let NextToken;
do {
const r = await ec2.describeInstances({
Expand All @@ -106,12 +121,12 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l
for (const i of res.Instances || []) {
const t = tagsOf(i.Tags);
const capacity = CAPACITY[t.capacity] || (i.InstanceLifecycle === 'spot' ? 'spot' : 'on-demand');
if (capacity === type) n += Number(t.vcpus) || DEFAULT_SHAPE.vcpus;
out.push({ id: i.InstanceId, job: t['job-id'], capacity, vcpus: Number(t.vcpus) || DEFAULT_SHAPE.vcpus, serving: ['pending', 'running'].includes(i.State.Name) });
}
}
NextToken = r.NextToken;
} while (NextToken);
return n;
return out;
}

// The tags a runner's instance or task carries: the job, for the swap
Expand Down Expand Up @@ -162,11 +177,11 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l
return arn;
}

// vCPUs of the cluster's tasks on one Fargate capacity, from the shape
// tags. A task counts until it is STOPPED: one told to stop (desired
// STOPPED) still holds its vCPUs while it deprovisions.
async function taskVcpusIn(type) {
let n = 0;
// The cluster's tasks, one record each, as instancesLive. A task holds its
// vCPUs until it is STOPPED: one told to stop (desired STOPPED) still holds
// them while it deprovisions, but is no longer serving its job.
async function tasksLive() {
const out = [];
for (const desiredStatus of ['RUNNING', 'STOPPED']) {
let nextToken;
do {
Expand All @@ -175,13 +190,14 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l
const d = await ecs.describeTasks({ cluster: env.CLUSTER, tasks: r.taskArns, include: ['TAGS'] });
for (const task of d.tasks || []) {
const t = tagsOf(task.tags);
if (task.lastStatus !== 'STOPPED' && CAPACITY[t.capacity] === type) n += Number(t.vcpus) || DEFAULT_SHAPE.vcpus;
if (task.lastStatus === 'STOPPED') continue;
out.push({ id: task.taskArn, job: t['job-id'], capacity: CAPACITY[t.capacity], vcpus: Number(t.vcpus) || DEFAULT_SHAPE.vcpus, serving: desiredStatus === 'RUNNING' });
}
}
nextToken = r.nextToken;
} while (nextToken);
}
return n;
return out;
}

async function launchInstance(m, name, runnerId, jitConfig) {
Expand Down Expand Up @@ -229,12 +245,29 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l
if (!j) return log(JSON.stringify({ ...job, dropped: 'job not found' }));
if (j.status !== 'queued') return log(JSON.stringify({ ...job, dropped: j.status }));

// One listing serves both checks. A job whose runner is still serving it is
// not launched again: its check goes on, so a runner that idles out without
// the job is replaced by a later check (#51).
const type = CAPACITY[m.label];
const machines = await (PROVIDER[m.label] ? tasksLive() : instancesLive());
const served = machines.find((x) => x.serving && x.job === String(m.job_id));
if (served) {
await sqs.sendMessage(checkLater(m));
return log(JSON.stringify({ ...job, served: served.id }));
}

// Launched before and not served now: wait out the gap before a relaunch.
const launches = Number.isInteger(m.launches) && m.launches > 0 ? m.launches : 0;
if (launches && now() - m.launched_at < relaunchGapMs(launches)) {
await sqs.sendMessage(checkLater(m));
return log(JSON.stringify({ ...job, waiting: `relaunch ${launches + 1} from ${new Date(m.launched_at + relaunchGapMs(launches)).toISOString()}` }));
}

// A soft budget: launches checked at the same moment may overshoot it. A
// job that would go over it goes back on the queue as a new message, so
// retries don't pile up towards the dead-letter queue.
const type = CAPACITY[m.label];
const budget = Number(budgets[type]);
const held = await (PROVIDER[m.label] ? taskVcpusIn(type) : vcpusIn(type));
const held = machines.filter((x) => x.capacity === type).reduce((n, x) => n + x.vcpus, 0);
const { vcpus } = shapeOf(m);
if (held + vcpus > budget) {
await sqs.sendMessage({ QueueUrl: env.QUEUE_URL, MessageBody: JSON.stringify(m), DelaySeconds: 60 });
Expand All @@ -258,6 +291,9 @@ function createHandler({ secret, fetch, ec2, ecs, sqs, now, env, log = console.l
await gh('DELETE', `/orgs/${org}/actions/runners/${r.runner.id}`, token).catch(() => {});
throw e;
}
// After the launch, so a failed send here never removes a live runner: the
// retry then finds it served and sends the check.
await sqs.sendMessage(checkLater({ ...m, launches: launches + 1, launched_at: now() }));
}

return async (event) => {
Expand Down
5 changes: 4 additions & 1 deletion test/launcher-ecs.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -73,12 +73,15 @@ const job = (label, vcpus, arch, extra = {}) => ({ Records: [{ messageId: 'm0',
}) }] });

test('ECS accepts the launcher\'s RunTask for every Fargate capacity, size and architecture', { skip }, async () => {
// One job id per launch: a task still serving a job is not launched again (#51),
// and the emulator keeps the tasks of earlier launches.
let jobId = 1000;
for (const [label, provider] of [['ci-fargate-spot', 'FARGATE_SPOT'], ['ci-fargate', 'FARGATE']]) {
for (const vcpus of [2, 4, 8, 16]) {
for (const arch of ['x64', 'arm64']) {
const what = `${label} ${vcpus} ${arch}`;
const { handler, logs } = launcher(fakeFetch());
assert.deepEqual(await handler(job(label, vcpus, arch)), { batchItemFailures: [] }, `${what}: ${logs.join('\n')}`);
assert.deepEqual(await handler(job(label, vcpus, arch, { job_id: jobId++ })), { batchItemFailures: [] }, `${what}: ${logs.join('\n')}`);
const arn = JSON.parse(logs.find((l) => l.includes('"launched"'))).launched;
const [task] = (await ecs.describeTasks({ cluster: 'ci-runner', tasks: [arn], include: ['TAGS'] })).tasks;
assert.equal(task.capacityProviderName, provider, what);
Expand Down
Loading
Loading