One CloudFormation stack, ci-runner, in zeroae-code / us-east-1.
make help lists the targets. The owner's procedures are in
runbook.md, and the traps in traps.md.
template.yaml source template (valid YAML, with inline markers)
functions/<name>.js one Lambda each: one CommonJS file, no npm deps
test/<name>.test.js node --test tests, run against functions/<name>.js
scripts/build-template.js template.yaml -> build/template.yaml (git-ignored)
- Write
functions/<name>.js. It must be self-contained: onlynode:*built-ins, plus@aws-sdk/*(shipped with thenodejs22.xruntime) required lazily insideexports.handler, never at the top of the file. The one exception is a localrequire('./<other>.js')on one line, which the build bundles in (the launcher bundlescloud-config.jsthis way). If the file setsmodule.exports = {...}, export the handler withmodule.exports.handler = ..., notexports.handler. - Put the logic behind a
createHandler(deps)factory that takes its adapters (AWS clients,fetch, clock) and returns the handler. Tests import the file and callcreateHandlerwith fakes.exports.handlerbuilds the real adapters once and delegates.functions/setup.jsis the example. - In
template.yaml, give it:Code: ZipFile: |whose only line is//@inline functions/<name>.js, indented as block content. The build replaces the pair with one line,ZipFile: "<the file as a JSON string>"(YAML reads a JSON string as the same text). That saves the block's indentation on every line. It also drops the file's whole-line//comments, but not inside template literals.Handler: index.handler,Runtime: nodejs22.x.- a
AWS::Logs::LogGroup/aws/lambda/${AWS::StackName}-<name>, wired in throughLoggingConfig; - its own role
PowerUserPB-${AWS::StackName}-<name>, withPermissionsBoundary: arn:aws:iam::aws:policy/PowerUserAccessand an inline policy of the same name granting only what that function needs. The one exception: a role that passes a role (iam:PassRole) takes the account'sZAEPowerUserBoundaryinstead, sincePowerUserAccessallows noiam:PassRole(see the launcher's boundary in "Launching for queued jobs").test/conventions.test.jschecks both.
- A Function URL with
AuthType: NONEneeds two permissions:lambda:InvokeFunctionUrl(FunctionUrlAuthType: NONE) andlambda:InvokeFunction(InvokedViaFunctionUrl: true).
make test:node --test test/*.test.js.make build: writesbuild/template.yamland prints its size against the 1 MB (1,048,576-byte) limit, the largest template CloudFormation accepts from S3. The build fails at or over it, andtest/build.test.jschecks it too. The build drops whole-line//comments from the inlined code and whole-line#comments from the template (outside block scalars), so comment the source freely.make deploy: test, build, thencloudformation deploywithCAPABILITY_NAMED_IAMand--s3-bucket $(BUCKET) --s3-prefix $(STACK), then print the outputs. It uploads the template to the deploy bucket and deploys from there, since a template sent inline must be under 51,200 bytes. It creates the stack the first time and updates it after. The bucket is made outside the stack, once per account (see "Deploy and update the stack" in the runbook);BUCKETdefaults to$(STACK)-deploy-<account id>, looked up withsts get-caller-identity. OverrideSTACK,PROFILE,REGION,BUCKETon the command line if needed.make validate: build, upload the template to the deploy bucket as$(STACK)/validate.yaml, and validate it from there.
GET on the SetupUrl output serves a page that auto-POSTs the App manifest
to GitHub. GitHub redirects back with code and state; setup exchanges the
code and writes app_id, app_slug, private_key, webhook_secret into the
${StackName}/github-app secret. Once the secret holds private_key, every
call answers 410.
state is <expiry>.<nonce>.<HMAC>, keyed by the state_key the secret is
created with, valid for 15 minutes, and must match a Secure; HttpOnly; SameSite=Lax cookie set by the GET, so a code is only accepted in the
browser that started the flow. Nothing is stored for it.
The stack has its own VPC (10.77.0.0/16): three public subnets in the
region's first three AZs (MapPublicIpOnLaunch), one route table with a
default route to an internet gateway, no NAT, no flow logs, no endpoints. The
security group RunnerSecurityGroup has no inbound rules.
RunnerLaunchTemplate (x64) and Arm64RunnerLaunchTemplate (arm64) hold
everything a CI runner instance shares, and differ only in the image: the AL2023
image for the architecture through resolve:ssm: (resolved at each launch:
al2023-ami-kernel-default-x86_64 or al2023-ami-kernel-default-arm64), 50 GB encrypted gp3
deleted on termination, IMDSv2 required with hop limit 1, shutdown behaviour
terminate, the tag ci-runner:stack=<stack> on instance and volume. It has
no instance profile, no instance type and no user-data.
A launch is an EC2 Fleet of type instant for one instance, from the job's
architecture's launch template: overrides for m8i/m8a/m7i/m6i/m6a/m7a (x64) or
m7g/m6g/m8g (arm64) at the job's size (.large, .xlarge, .2xlarge or
.4xlarge for 2, 4, 8 or 16 vCPUs) in each of the three subnets, Spot
capacity-optimized for ci-spot. Every family is offered at every size in
all three of the stack's AZs (use1-az1, use1-az2, use1-az4; checked
2026-11-13 with aws ec2 describe-instance-type-offerings --location-type availability-zone), so no family/AZ pair is left out. Check again before
adding a family or moving the subnets. CreateFleet can't override user-data, so
each launch first adds a launch template version (source $Default, only
UserData), launches from it, and deletes that version again. The outputs
LaunchTemplateId, Arm64LaunchTemplateId, SubnetIds and SecurityGroupId
are what a launch needs.
functions/cloud-config.js exports renderCloudConfig({ jitConfig, idleSeconds = 900 }), which returns the #cloud-config document. It has no
require(), so the launcher can bundle it.
test/cloud-config.test.js covers it. On the instance:
bootcmdcreatesdockerwith gid 123 before thedockerpackage does.write_filesputs the JIT config in/etc/ci-runner/jitconfig(0600) and the start script in/usr/local/sbin/ci-runner.runcmdstarts Docker and runs the start script, which pullsghcr.io/zeroae/ci-runner:latest, copies the image's/home/runner(the runner itself) onto the host, and runs the image with the host's socket,/tmpand/home/runnermounted at the same paths (the host's/tmpis Amazon Linux's tmpfs, so a root-disk directory is bind-mounted over it first), as./run.sh --jitconfig. A job'sdocker run -vgoes to the host's daemon, so its source must be a host path; sharing those two gives a job the same paths as a GitHub-hosted runner. If the container's log has noRunning jobline afteridleSeconds, the script stops the container. When the container exits, the script'sEXITtrap runsshutdown -h now, which terminates the instance.
A runner stopped by the idle exit stays registered, offline, until the sweeper (or GitHub, for ephemeral runners) removes it.
make hand-launch (scripts/hand-launch.sh) mints an org JIT config for the
ci-runner group with labels self-hosted, CAPACITY, SIZE and ARCH
(all required; none has a default) through the owner's gh, which needs
admin:org. It then renders the user-data and launches one instance of that
shape as above, with the launcher's fleet options: Spot capacity-optimized
for CAPACITY=ci-spot, On-Demand lowest-price for ci-ondemand. It is
tagged with its shape (make hand-launch CAPACITY=ci-spot SIZE=ci-cpu-8 ARCH=x64 launches an x64 Spot .2xlarge; ARCH=arm64 a Graviton from the
arm64 launch template). It refuses ci-fargate-spot and ci-fargate, which
run a task, not an instance: the runbook's "Run a task by hand" does that.
It prints the instance id. IDLE_SECONDS=120 shortens the
idle exit for a test. make runners lists the stack's instances that are not
terminated.
functions/sweeper.js runs every 10 minutes (an EventBridge rule). Each run:
- Reads the secret. Once it holds
private_key, it signs an App JWT (RS256,iss=app_id), gets the org installation's token, and lists the runners in theci-runnergroup. Before the App exists it skips this. - Describes the instances tagged
ci-runner:stack=<stack>, and terminates thosepending,running,stoppingorstoppedand launched over 2 hours ago. - Lists the cluster's tasks whose desired status is
RUNNING(not yet told to stop), with their tags, and stops (StopTask) those created over 2 hours ago. Every task in the stack's cluster counts, a hand-run one too. - Removes each runner in the group that is
offline, not busy, and has no instance or task (see below). A GitHub failure doesn't stop steps 2 and 3, but fails the run; an ECS failure fails it after step 2, before any registration is removed.
Runners are listed before instances and tasks are, so a runner whose
machine launches in between is still seen with it. The run returns
{terminated, stopped, removed}.
Its role allows GetSecretValue on the secret, ec2:DescribeInstances, and
ec2:TerminateInstances only on instances whose aws:ResourceTag/ci-runner:stack
is the stack's name. A TerminateInstances naming an untagged instance, or
another stack's, is denied. On ECS it allows ecs:ListTasks with the
condition ecs:cluster = the cluster, and ecs:DescribeTasks/ecs:StopTask
only on the cluster's tasks (task/<cluster>/*).
Whatever launches a runner (scripts/hand-launch.sh, the launcher) must:
- tag the instance or task
ci-runner:runner=<runner name>, the name it gavegenerate-jitconfig. It may also tagci-runner:runner-id=<runner id>; either tag matching keeps the registration. - put a UTC
YYYYmmddHHMMSStimestamp of the mint in the name as its own--separated token, for exampleci-runner-20261112113000-<job id>. The registration exists before its instance does, so the sweeper leaves a runner alone for 10 minutes after that time. A name with no stamp gets no grace.
Every instance not yet terminated counts, so a runner whose instance is
shutting down is removed on a later run. A task counts until it is told to
stop: the sweeper lists only tasks whose desired status is RUNNING, so a
runner whose task the sweeper stops is removed on the next run. (ECS itself
keeps a stopped task's tags readable through DescribeTasks.)
webhook (functions/webhook.js, the WebhookUrl Function URL, which the
App's webhook points at), for each delivery:
- Verifies
X-Hub-Signature-256against the secret'swebhook_secret(401otherwise;503before the App exists). - Ignores (
200) any event butworkflow_job. - Fence 3: refuses (
403, logged) unlessrepository.visibilityisprivateand the owner isGITHUB_ORG. - On
in_progress, logs a runner running another job than its own (below) and answers. - On
completed, tells the notifier when the job succeeded on a runner of this stack (below) and answers. - Ignores (
200) actions other thanqueued, and jobs outside the label contract (case-insensitive):self-hosted, exactly one ofci-spot/ci-ondemand/ci-fargate-spot/ci-fargate, exactly one size label (ci-cpu-2,-4,-8,-16), exactly one ofx64/arm64, and optionallylinux. None has a default: a missing size or architecture label, an unknownci-cpu-*, or bothx64andarm64, are outside it. A job label (below) is required on top, and checked once the rest passes: a job with none is ignored (200,ignored: labels) and logged as{"ignored": <repo>, "job_id", "why": "no job label"}. A job with two, or one that leaves out its run and attempt, names another run or attempt, has nothing after them, or is over 255 characters, is ignored (200,ignored: job label) and logged as{"ignored": <repo>, "job_id", "label", "why"},whybeingtwo job labels,job label without its run and attempt,job label of another run,job label of another attempt,job label names no joborjob label over 255 characters. - Enqueues
{repo, job_id, run_id, head_sha, job_name, label, vcpus, arch, labels, installation_id}(labelis the capacity label, lower case;vcpusis 2, 4, 8 or 16;archisx64orarm64) and answers202.
The job label (ADR-0004). A job must carry one label unique to it,
ci-job-<run_id>-<run_attempt>-<job key>-<matrix index>, compared
case-insensitively. The webhook accepts it only when it starts with
ci-job-<run_id>-<run_attempt>- of the event's workflow_job and has
something after that. It goes into the message's labels with the rest, the
launcher mints the runner's JIT config with those labels, and so the runner
launched for the job can take no job with another job label. GitHub gives a
job to a runner whose labels include all of the job's, so a job without one
could take any runner of its shape: that is why the label is required, and
why a job without one, though ignored, can still swap while it waits.
A workflow writes it in runs-on, with the job key typed by hand: inside
runs-on, github.job evaluates to an empty string. strategy.job-index is
0 in a job with no matrix. Measured on 2026-11-20 in a scratch repo, by
reading the queued jobs' labels back from the API:
jobs:
tests:
runs-on: [self-hosted, ci-spot, ci-cpu-4, x64, "ci-job-${{ github.run_id }}-${{ github.run_attempt }}-tests-${{ strategy.job-index }}"]A runs-on built by an expression (the adoption variable's fallback to a
GitHub-hosted runner) builds the label with format:
runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON(format('["self-hosted","ci-spot","ci-cpu-4","x64","ci-job-{0}-{1}-tests-{2}"]', github.run_id, github.run_attempt, strategy.job-index)) || 'ubuntu-latest' }}A reusable workflow (workflow_call) sees only its own job keys and its own
strategy, so a run that calls it twice, or from a matrix, gives both calls
the same label. Give it an input the caller makes unique, and put it in the
label:
# the called workflow
on:
workflow_call:
inputs:
call: { type: string, required: true }
jobs:
e2e:
runs-on: ${{ fromJSON(format('["self-hosted","ci-spot","ci-cpu-4","x64","ci-job-{0}-{1}-e2e-{2}-{3}"]', github.run_id, github.run_attempt, strategy.job-index, inputs.call)) }}
# the caller
jobs:
e2e-a:
uses: ./.github/workflows/e2e.yml
with: { call: e2e-a }A caller with a matrix passes ${{ strategy.job-index }} in the input too.
GitHub refuses a runner label of 257 characters or more and silently cuts a
256-character one to 255, so the runner would not carry the job's label
(measured 2026-11-20 with generate-jitconfig); the webhook ignores a job
label over 255 characters. A 98-character job key is
accepted and a 100-character one is a workflow file error, so with an 11-digit
run id and a 3-digit matrix index a label stays under 130 characters, plus the
input.
Telling the notifier a job succeeded. On workflow_job completed with
conclusion success, and a runner_name that is ${stack}-<14-digit stamp>-<job id> (a job on a GitHub-hosted runner, another stack's, or none is
ignored, 200), the webhook invokes the notifier asynchronously
(lambda:InvokeFunction on it, env NOTIFIER_FUNCTION) with {jobSucceeded: {repo, job_id, run_id, head_sha, job_name, installation_id}} and answers
202. The App's workflow_job subscription already delivers every action
(about 3 deliveries per job: queued, in_progress, completed), so nothing
about the App changes. The webhook does no GitHub call itself.
A runner running another job (ADR-0004). A runner can take only a job
whose labels it carries, and each carries its own job's label, so no job
with a job label can take another job's runner. A job of our shape without
one gets no runner of its own, but can still take one launched for another
job: swaps end only once every such job carries its label. On
workflow_job in_progress the
webhook still reads runner_name: a name that isn't ${stack}-<14-digit stamp>-<job id> (a hand launch, a GitHub-hosted runner, another stack), or
whose job id is the event's, is ignored (200, ignored: no swap).
Otherwise it logs {"anomaly": "runner of another job", "runner", "launched_for": <job id>, "took": {repo, job_id}} and answers 200. It
re-queues and re-tags nothing. The job the runner was launched for is not
stranded: the launcher's next check finds that runner busy while the job is
still queued, and launches it another runner at once (below). A reclaim of
the swapped machine is not reported (the notifier finds its launch job not
running on it). Until ADR-0004 the webhook repaired swaps instead (ADR-0003).
Its role allows GetSecretValue on the secret, SendMessage on the queue and
lambda:InvokeFunction on the notifier, and nothing on EC2 or ECS.
The queue ${stack}-jobs has a 120 s visibility timeout. A message that
fails 5 receives goes to ${stack}-jobs-dlq (kept 14 days). The outputs are
JobQueueUrl and JobDeadLetterQueueUrl.
launcher (functions/launcher.js, fed one message at a time, reporting
failures per message), for each job:
- Drops a malformed message, or one whose repo isn't in
GITHUB_ORG, or whosevcpusorarchisn't served. A message with neither (enqueued before the size labels) gets 4 vCPUs andx64. - Signs an App JWT and gets an installation token for the job's installation. The token is cached until 5 minutes before it expires.
GET /repos/{repo}/actions/jobs/{job_id}: anything butqueued(or a404) is dropped, and the message is consumed.- The vCPU budget: sums the vCPUs of the stack's instances
(
ci-runner:stacktag) that arepending,running,stoppingorstoppedon the job's capacity, both architectures together, from their shape tags (ci-runner:capacity,ci-runner:vcpus). An instance launched before those tags counts as 4 vCPUs, Spot if it hasInstanceLifecycle: spotand On-Demand otherwise. If the job's vCPUs would take the sum over the capacity's budget (SpotVcpuCap, default 24;OnDemandVcpuCap, default 8), it sends the message to the queue again with a 60 s delay and consumes the original, so waiting doesn't count towards the dead-letter queue. Nothing is minted. A job that brings the sum exactly to the budget launches. It's a soft budget: messages checked at the same moment can overshoot it. A shutting-down instance no longer counts. A Fargate job counts the cluster's tasks instead (see "The Fargate capacities"). - Looks up the
RunnerGroupparameter's group id by name (cached), and mints a JIT config with the job's labels. The runner is named${stack}-<UTC YYYYmmddHHMMSS>-<job id>, as the sweeper requires. - For a Fargate job, runs one task (see "The Fargate capacities"). Otherwise
renders the
#cloud-configand launches one instance as in "Launching one by hand": Spotcapacity-optimizedforci-spot, On-Demandlowest-priceforci-ondemand, at the job's size, from the launch template and the families of the job's architecture. Tags:ci-runner:runner,ci-runner:runner-id,ci-runner:label,ci-runner:repo,ci-runner:job-id,ci-runner:run-id,ci-runner:head-sha,ci-runner:job-name,ci-runner:labels(the job's labels, comma-joined), and the shape:ci-runner:capacity(the capacity label),ci-runner:vcpusandci-runner:arch; plus the launch template'sci-runner:stack. Thelaunchedlog line carrieslabel,vcpusandarch. - If no instance or task launches, it removes the registration it just minted and fails the message, which comes back 120 s later.
The check that follows (#51). Every launch, and every check that finds the
job still queued and served, sends the job's own message again, delayed 300 s
(RECHECK_SECONDS), once the launch is done. A check first looks at the
machines this stack tags with the job (ci-runner:job-id): a pending or
running instance, or a task not yet told to stop. If one is serving the job,
the check sends its own message on and launches nothing (its log line says
served). A machine whose runner is busy on GitHub (GET /orgs/{org}/actions/runners/{ci-runner:runner-id}) while the job is still
queued is not serving it: its runner took another job, a swap (ADR-0004). When
every machine of the job is so, the check reads the job again (the runner may
have just taken it, which drops the message) and, still queued, launches it at
once, with no gap, its log line carrying swapped (the machines' ids). The
next check's message remembers those ids as swapped, so a machine still busy
on a later check (the relaunch's runner gone, the job still held) waives the
gap only the first time. A
machine with no ci-runner:runner-id tag, or whose runner GitHub no longer
has (404), counts as serving. If none is serving, the job is launched as
above, but no sooner than 15
minutes after its last launch, a gap that doubles for each launch after that, up
to 60 minutes (its log line says waiting until then). So a job whose runners
never take it (the repo left the group, labels nothing matches) costs about one
launch an hour, not one every check. The message carries launches and
launched_at for this; one without them (the webhook's) has
had no launch and launches at once. The checks end when GitHub
no longer has the job queued, which drops the message. This is why a job held
behind a concurrency group the first job holds gets a runner once the group
frees: a runner that idled out first is replaced at the next check. The sweeper
does not do this, since it has no list of queued jobs. Not yet checked live:
the behaviour is covered with fakes in test/launcher.test.js.
There is no de-duplication on job id beyond that check. A duplicate delivery launches a second runner at worst, and the idle exit and the sweeper remove it.
cloudformation deploy keeps a parameter's previous value unless it's
overridden, so after lowering a budget for a test, restore it explicitly
(--parameter-overrides SpotVcpuCap=24 OnDemandVcpuCap=8).
The launcher's role allows GetSecretValue, the queue's receive, delete and
send, ec2:DescribeInstances (for the budgets),
Create/DeleteLaunchTemplateVersions on the two launch templates, and
CreateFleet/RunInstances pinned to the two launch templates, the three subnets
and the security group. It also allows CreateTags only while creating
(ec2:CreateAction), and ssm:GetParameters on the public AL2023 parameters,
which the launch template's resolve:ssm: image needs. For Fargate it
allows ecs:RunTask on the two task definitions in the stack's cluster
(ecs:cluster), iam:PassRole on the execution role only and only to
ecs-tasks.amazonaws.com, ecs:TagResource only while creating
(ecs:CreateAction: RunTask), and ecs:ListTasks/ecs:DescribeTasks on the
cluster (for the budgets). It has no other iam:PassRole. All four conditions
hold live (checked 2026-11-14 with one job on each Fargate capacity).
Its permissions boundary is the account's ZAEPowerUserBoundary, not
PowerUserAccess like every other role: RunTask passes the task
definition's execution role, and PowerUserAccess allows no iam:PassRole
at all. ZAEPowerUserBoundary is PowerUserAccess plus iam:PassRole on
PowerUserPB* roles; the policy above narrows that to the execution role.
test/conventions.test.js requires that boundary on a role that passes one,
and PowerUserAccess on every other. CreateFleet's check on the instances, volumes and network
interfaces it creates carries no launch-template key, so those resources
are granted to CreateFleet without one. RunInstances keeps the condition.
ci-fargate-spot (Fargate Spot) and ci-fargate (Fargate On-Demand) run a
job in a Fargate task: no instance to boot, no host to clean up, and no
Docker daemon (ADR-0001). The template's Fargate resources:
RunnerCluster, named after the stack, with theFARGATEandFARGATE_SPOTcapacity providers and no default strategy.RunnerTaskDefinition(x64, family${stack}-runner) andArm64RunnerTaskDefinition(arm64,${stack}-runner-arm64): two, becauseRunTaskcan't override the platform. They differ only inCpuArchitecture:awsvpc, 4 vCPU / 16 GB (each launch overrides both), 50 GB of ephemeral storage, one essential containerrunnerfrom${ImageRepositoryUri}:latest(ECR, lazy-loaded through its SOCI index), entrypoint/usr/local/bin/ci-runner-idle-exit,InitProcessEnabled, andawslogstoRunnerTaskLogGroupwith the stream prefixrunner. They have no task role, so a job finds no AWS credentials: noAWS_CONTAINER_CREDENTIALS_RELATIVE_URI, and the instance metadata service isn't reachable from Fargate. The task metadata endpoint (ECS_CONTAINER_METADATA_URI_V4) is there, and holds no credentials.RunnerExecutionRole,PowerUserPB-${stack}-runner-exec, which ECS uses to start a task and the job never sees:ecr:GetAuthorizationToken, the pull calls onImageRepositoryonly, and writing to the log group.RunnerTaskLogGroup,/ecs/${stack}-runner, kept 7 days: the runner's output (Listening for Jobs,Running job), one stream per task (runner/runner/<task id>).
The outputs ClusterName, TaskDefinitionArn and Arm64TaskDefinitionArn
are what a hand-run task needs.
The launch. For a Fargate job the launcher calls RunTask with:
- the capacity provider (
FARGATE_SPOTorFARGATE), weight 1, and the task definition of the job's architecture; - task-level
cpuandmemoryfor the size (ci-cpu-8:8192and32768, so 4 GB per vCPU as on EC2); - a container override: the command
./run.sh --jitconfig <config>, and the environment below. The JIT config is readable toecs:DescribeTasks, the same exposure as an instance's user-data; assignPublicIp: ENABLEDin the stack's subnets and security group. arm64 leaves out any subnet inuse1-az3, where Fargate has no arm64 (the template passes each subnet's AZ id asSUBNET_AZ_IDS; the stack's subnets are inuse1-az1,-az2and-az4today, so nothing is left out);- the instances'
ci-runner:*tags, plusci-runner:stack. ECS allows only letters, digits, spaces and_ . : / = + - @in a tag, so on a taskci-runner:labelsis joined by spaces, not commas, and any other character (a job name's brackets or commas) becomes_.
The launch is logged as launched: <task ARN>. A RunTask that returns no
task fails the message with no task launched: <reason>: <detail>.
The environment. Inside a task, nproc can report the host's CPUs (8
on a 4-vCPU task in 3 of the spike's 4 tasks) and there is no cgroup
cpu.max, so tools that size their parallelism by it oversubscribe. The
container override sets MAKEFLAGS=-j<n> (which also reaches CMake's Makefile
generator) and CI_RUNNER_VCPUS=<n> to the task's vCPUs; the runner's
environment reaches every step. From CI_RUNNER_VCPUS the image's entrypoint
writes /home/runner/.cargo/config.toml with [build] jobs = <n> (an existing
file is left alone, and logged). Cargo reads .cargo/config.toml from every
ancestor of its working directory, so it applies to every checkout under
/home/runner/_work, whatever CARGO_HOME a job sets.
No CARGO_* or CMAKE_* variable is set, on purpose: Swatinem/rust-cache
hashes every variable starting CARGO, CC, CFLAGS, CXX, CMAKE or
RUST into its cache key
(src/config.ts),
so a CARGO_BUILD_JOBS (as the first version set) gave Fargate jobs their own
key, and they never hit the cache an EC2 job saved. Its hash of
.cargo/config.toml files looks only inside the workspace, so the home one
doesn't change the key. IDLE_SECONDS is passed on too, when the launcher has
it (a test).
The idle exit. image/ci-runner-idle-exit runs its arguments in their own
session, copying their output to a file, and stops them if that output has no
Running job line after IDLE_SECONDS (default 900). A stop is a TERM to every
process in the session (pkill -s), then a KILL after IDLE_GRACE_SECONDS
(default 30, as docker stop -t 30 gives an EC2 runner). The session matters:
run-helper.sh and Runner.Listener run in their own process group, so a TERM
to run.sh's group alone misses them and the task never stops (seen live).
With the whole session signalled, the runner exits at once, status 143. It passes a TERM or INT on (a StopTask), and exits with the
runner's status, so the essential container exits and the task stops. The
image sets no ENTRYPOINT, so EC2's host script, which runs ./run.sh with
its own idle exit, is unchanged; only the task definitions use the wrapper.
image/smoke.sh checks it with fake runners and a 2 s timeout. For a test,
IDLE_SECONDS can be set per task in a container override, as in the
runbook's "Run a task by hand".
The budgets. FargateSpotVcpuCap (default 48) and FargateVcpuCap
(default 8) budget the two capacities, both architectures together, as the
EC2 budgets do. Fargate Spot's default is 48, double the others: at 24, one
consumer's 9-job matrix at ci-cpu-8 ran in waves and spent most of its
queued time on budget waits, and the Fargate Spot quota (140) has room. The launcher lists the cluster's tasks not yet STOPPED:
those whose desired status is RUNNING (provisioning, pending or running),
and those told to stop (desired STOPPED) whose last status isn't STOPPED
yet, since a deprovisioning task still holds its vCPUs. It sums their
ci-runner:vcpus on the job's capacity (ci-runner:capacity). A task with no
vCPU tag counts as 4. EC2 instances and tasks never count towards each
other's budgets.
A task's lifecycle is an instance's: the sweeper stops it 2 hours after launch and keeps its runner's registration while it lives, and the notifier reports a Fargate Spot reclaim (below).
functions/notifier.js runs on two rules:
${stack}-notifier: every EC2 Spot Instance Interruption Warning in the region (an event pattern can't match on tags);${stack}-notifier-tasks: every ECS Task State Change of the stack's cluster withstopCode: SpotInterruption. A reclaimed task's changes all carry it, from the warning (desiredSTOPPED, stillRUNNING, about 2 minutes before the stop) toSTOPPED. The notifier reads the tags withDescribeTasks, which still returns them after the stop: measured live on 2026-11-14, a task's tags stayed readable fromStopTaskthroughSTOPPEDand minutes after, whileListTagsForResourcerefused the task (The specified task is stopped) from the moment it was told to stop. So every change can be read, and the check-run lookup keeps it to one report per job (aws-emulator, unlike AWS, drops the tags at the stop).
For each event it:
-
Describes the instance, or the task (
DescribeTaskswith its tags; the event carries none), and ignores it (logged) unless it carriesci-runner:stack=<stack>,ci-runner:head-sha,ci-runner:job-idand aci-runner:repoinGITHUB_ORG: the launcher's tags. An instance frommake hand-launchor a task run by hand has no job tags and is ignored, as is a task whose tags are gone. -
Signs an App JWT, finds the repo's installation (
GET /repos/{repo}/installation) and gets its token. -
Reads the job from GitHub (
GET /repos/{repo}/actions/jobs/{job_id}), the job the tags name: the one the machine was launched for, the only one its runner can take (ADR-0004). It reports only a job the machine's CI runner is running:statusin_progressandrunner_nameequal to the machine'sci-runner:runnertag. Otherwise it logsignored <id>: its runner is not running job <id> (...)and posts nothing: the job isqueued, orcompleted, or another CI runner runs it. The usual case is an idle CI runner: its job was cancelled, or another launch's runner for the same job (a relaunch, #51) took it. -
Takes the job's real name from that job, falling back to the tag: a tag is cut at 256 characters, and a task's has
_for brackets and commas. The webhook's word (step 8) carries GitHub's name too, so the two meet. -
Skips the job if the commit already has its reclaim check run (same name,
external_idthe job id, the title below or the survived title), so a task's later changes and a repeated event report nothing more. -
Creates a check run on the head sha: name
ci-runner / <job name>(orjob <id>with no name),completed, conclusionfailure, titleSpot capacity reclaimed: re-run this job, a summary naming the job and the instance or task, linking the run, and telling a person to re-run a failed job or, for a job GitHub keeps in progress, to force-cancel the run first (its command, with the run's id, and the warning that it ends every job still running in the run; see "Read a lost job" indocs/runbook.md), anddetails_urlthe job's page. The title must not change: steps 5 and 8 find a report by it. An EC2 reclaim and a Fargate one give the same check run, but for the summary's machine. -
Reads the job once more after posting: a job that finished with
successwhile this ran is cleared as in step 8. -
Clears a report the job survived. AWS warns 2 minutes before it takes the machine, and a job can finish inside that. A job that finishes with
successwas not lost, so its reclaim check run changes toneutral, titledThe job finished despite the Spot reclaim warning, and the notifier logssurvived: <repo> <job name>. Three paths clear a report:- the webhook's word (above): the notifier gets the token (the event
carries the installation id), lists the commit's check runs named
ci-runner / <job name>, and clears each reclaimfailurewhosedetails_urlis under the job's run. That takes in the report of an earlier attempt of the job: a re-run's job has a new id, but the same run. With none to clear it logsnothing to clearand changes nothing; - a retry or a later change of a reclaim event that finds its own
standing
failure(step 5) for a job that has since finished withsuccess; - step 7, for the word that came before the report existed.
A job that ends with any other conclusion keeps the
failure. If the webhook's invoke of the notifier fails, the delivery fails (502): redeliver it from the App's advanced settings, or the retry and later-change paths clear the report. - the webhook's word (above): the notifier gets the token (the event
carries the installation id), lists the commit's check runs named
A GitHub failure fails the invocation, and EventBridge's async invoke retries it. Two events handled at the same moment can still report twice.
The launcher, the notifier and the sweeper each carry their own copy of
appJwt() and gh(), because bundling a shared module costs more template
bytes than the copies. test/conventions.test.js fails if the copies differ,
so change all three at once.
Its role allows GetSecretValue on the secret, ec2:DescribeInstances, and
ecs:DescribeTasks only on the cluster's tasks.
Fargate lazy-loads an image only from private ECR, through a SOCI index
(ADR-0002), so image.yml pushes the image to the stack's own ECR repo as
well as to GHCR. EC2 runners keep pulling ghcr.io/zeroae/ci-runner:latest:
an instance has no instance profile, so it can't pull from private ECR.
ImageRepository, named after the stack (ci-runner). Its lifecycle rule keeps the last 4 tagged images. Each build tags one image index, under:latestand:<date>-<sha>, so that's the last 4 builds. The index's children (each architecture's image manifest and SOCI index) are untagged, and no rule touches them: an untagged rule could expire the children of an index still in use.EmptyOnDeletelets a stack delete remove it, images and all.GitHubOidcProvider, GitHub's OIDC provider, created only when theExistingOidcProviderArnparameter is empty (theCreateProvidercondition; the pattern is zeroae/foresight'sgithub-deploy-zae). An account holds one provider per URL, so in an account that already has one, pass its ARN instead.ImagePushRole,PowerUserPB-${stack}-image-push, trusted only forsts:AssumeRoleWithWebIdentityfrom that provider withaudsts.amazonaws.comandsubexactlyrepo:zeroae@11018406/ci-runner@1455021495:ref:refs/heads/main. The repo uses GitHub's immutable subject, so thesubnames the owner and repo by id (gh api repos/zeroae/ci-runner/actions/oidc/customization/subshows the prefix); the name-only formrepo:zeroae/ci-runner:...never matches. Any other branch, a pull request, a tag or another repo carries a differentsub, and can't assume it. It allowsecr:GetAuthorizationToken(which has no resource-level permissions) and, on the repo only, the layer upload calls,PutImage,BatchGetImageandGetDownloadUrlForLayer.
The outputs ImageRepositoryUri and ImagePushRoleArn are the repo
variables ECR_IMAGE and AWS_IMAGE_PUSH_ROLE_ARN that image.yml reads.
There is nothing to rotate: the push uses a short-lived OIDC token, and no
AWS key or GitHub token is stored anywhere.
image.yml's soci job, on every branch:
- downloads each architecture's build as an OCI layout and merges both into one OCI image index, each manifest with the platform its config names;
- runs
soci convert --standalone --all-platforms(thesociCLI from awslabs/soci-snapshotter, releaseSOCI_VERSION, sha256-checked). It needs no containerd. The result is a SOCI-enabled image (SOCI index manifest v2): an index holding both image manifests and a SOCI index (artifactType: application/vnd.amazon.soci.index.v2+json) for each, and the job fails unless both are there; - pushes it with
skopeo copy --all --preserve-digeststo a localregistry:3service and reads it back, so a branch exercises the push.
On main, with both variables set, it then assumes the push role and
pushes the same index to ECR under :<date>-<sha> and :latest. Without
the variables (before the stack has the repo) those two steps are skipped
and the GHCR publish is unaffected.
image.yml publishes, to GHCR and to ECR, only from main, and the push
role trusts only main, so a change under image/ reaches no runner until
it is on main. Make an image/ change its own commit, and land it on
main before anything that needs it live. An integration branch that
carries such commits (cherry-picked to main, then merged back) lands with
a merge commit: a rebase would re-resolve every one of those conflicts.