Skip to content

fix(gateway): A client whose clock runs behind accepts a certificate the Gateway just issued - #36

Merged
sodre merged 5 commits into
mainfrom
feat/24-fargate-harness
Nov 20, 2026
Merged

sodre merged 5 commits into
mainfrom
feat/24-fargate-harness

Conversation

@sodre

@sodre sodre commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Closes #24

Summary

A client whose clock runs slightly behind the Gateway's now accepts a certificate the Gateway has just issued: the sidecar CA and every leaf start their validity one minute before they are issued, and a leaf never starts before the CA that signs it. Maintainers also get a harness, in qualification/fargate, that runs the Gateway's containment test and its runtime test on real ECS Fargate 1.4.0, on x86-64 and arm64.

qualification/fargate/
├── stack.yaml   # ECR repository, small public VPC, cluster, log group, bounded execution role, two task definitions
├── Dockerfile   # the Gateway (release build), both tests and what they need; cross-compiled, no emulation
├── qualify.sh   # the task's command: host facts, "namespaces are refused", then both tests
├── run.sh       # deploy | build | run | local | teardown
└── README.md

The harness found the certificate bug on its first run, and the owner chose to widen this ticket and fix it here, although spec #21 lists changes to the issuer as out of scope. On Fargate's kernel, time() advances once per 10 ms tick. The Gateway dated a leaf at the current second, so curl in the same task, reading time(), rejected a leaf issued within 10 ms after a second began as not yet valid. The runtime test hit that at the same step on every run, on both architectures.

 sign_leaf
-  notBefore = now
+  notBefore = max(now - 1 minute, the CA's notBefore)

Choices the ticket left open:

  • Region. The stack runs in us-east-2: us-east-1 was at its VPC limit while the prototype stack still stood. AWS_REGION chooses, and the README says so.
  • Release build. The image holds a release build, which is what operators run. CI's container job tests a debug build.
  • Base image. The last stage is python:3.12-bookworm, which already has Python and curl. The two other needs (the cryptography wheel and libnss_wrapper.so) are fetched for the target architecture and unpacked, so neither image needs emulation.
  • A run names its revision. run.sh run fails a task whose image was built from another commit than the checkout.

Evidence

Criterion Ticked Shown by
The harness (stack template, image build and a run script) lives in the repo outside the product code, with a short README yes qualification/fargate/run.sh run (the final run, below), which deployed, built and ran from those files
On Fargate 1.4.0 X86_64, the worker sandbox containment test passes under the filter sandbox yes qualification/fargate/run.sh run: 25 passed, 0 failed
On Fargate 1.4.0 X86_64, the runtime test passes under "filter", with the test grant authority and mTLS site inside the task yes qualification/fargate/run.sh run: sidecar and shared mTLS passed, filter-killed grant worker fenced, exit code 0
On Fargate 1.4.0 ARM64, the containment test passes yes qualification/fargate/run.sh run: 23 passed, 0 failed
On Fargate 1.4.0 ARM64, the runtime test passes yes qualification/fargate/run.sh run: sidecar and shared mTLS passed, filter-killed grant worker fenced, exit code 0
The run's output (kernel, Landlock ABI, layers logged, test results) is recorded on this ticket yes gh issue comment 24: the comment of 2026-11-20 holds the table and both task logs
#18 gets a comment asking the reporter to qualify, linking the build to try no Not run: Gateway images are published only for a release tag, so the build to link exists after the owner's next release
The prototype stack countersign-isolation-probe-18 (us-east-1) is deleted once the harness replaces it yes aws cloudformation delete-stack and wait stack-delete-complete, exit 0; describe-stacks then answered "does not exist"
Test Red Green
The harness fails when a test fails First Fargate run, before the fix: both tasks exited 1 with AssertionError at the runtime test's last phase, and run.sh run exited 1 ok
A client whose clock runs behind accepts the CA and the leaf (tests/sidecar.rs) Against the old issuer: panicked, Validity { not_before: 2026-11-20 6:18:07 ...}, the second of issuance ok
A client half a minute behind finds the shared issuer's leaf valid (local_shared_ca_checks_key_and_issues_for_tls) Against the old issuer: test result: FAILED. 0 passed; 1 failed ok
A leaf never starts before the CA that signs it (shared_issuer.rs) With the backdating unclamped: left: 1795156239, right: 1795156294 ok
run.sh run refuses a task built from another revision Images from af961f2, checkout at 30bedc2: did not run this checkout (v0.2.0-10-g30bedc2): build its image first, exit 1 ok
qualify.sh refuses an image without unshare With unshare off the PATH: The qualification image has no unshare., exit 1 ok

The final run, qualification/fargate/run.sh run at 30bedc2 in us-east-2, exited 0. Both tasks: Fargate Linux 1.4.0, kernel 6.1.190, user namespaces refused, no bubblewrap, startup line Gateway worker sandbox "filter": non-dumpable, Landlock ABI v2, seccomp allow-list, exit code 0. Two earlier runs at af961f2 passed the same way.

The cause was measured on Fargate with an instrumented copy of the runtime test: curl started at second .984, the leaf's notBefore was the next second, the coarse clock read about 10 ms behind the real one, and curl exited 35.

Also run, all exit 0: cargo test --locked -p countersign-gateway (lib 4 passed, bin 5 passed, sidecar 6 passed, upstream 1 passed, 0 failed, 0 ignored), cargo clippy -p countersign-gateway --all-targets -- -D warnings, scripts/filter-sandbox-container.sh (CI's container job, every step), runtime.py under the namespace sandbox on the host, and qualification/fargate/run.sh local.

Not run: the comment on #18, since Gateway images are published only for a release tag, so the build to link exists after the owner's next release. The rest of the workspace's suites (pnpm, the other crates), which CI runs: the change touches only the Gateway crate.

Merge Danger

Door: two-way. A revert restores certificates that start at the second of issuance; nothing stored changes.

Blast Radius: every certificate the Gateway issues itself.

The next Gateway release carries the issuer change: the sidecar CA and each leaf from the sidecar CA or a local shared CA are valid from one minute before issuance. Leaves from AWS Private CA are unchanged. The merge deploys nothing. The stack countersign-fargate-qualification was deleted from us-east-2 after the final run (run.sh teardown); run.sh deploys it again for the next qualification. No flag.

Reviews

Correctness, Standards, Spec and Security, two rounds: 15 findings, each with its outcome

Round 1 pinned 9a0cb18, round 2 30bedc2.

Correctness, round 1

  1. One aws ecs wait tasks-stopped gives up after ten minutes, which a hung task outlasts, and the script then died before printing its log. Fixed: it waits twice and prints the log and exit code either way.
  2. run-task exits 0 when Fargate places no task, and the reason was discarded. Fixed: the script prints the failure's reason and stops.
  3. local -A needs bash 4, which a stock Mac lacks. Fixed: an indexed array.
  4. A backdated leaf could start before a freshly minted shared CA. Fixed: the leaf's start is clamped to its CA's, with a test.
  5. get-log-events returns one page, and its failure was swallowed. Fixed: filter-log-events, which follows the pages, and a named line when the task left no log.
  6. run without build could pass on an image from another commit. Fixed: a task has to name the checkout's revision.
  7. The execution role's name has no region, so one stack name cannot deploy in two regions. Pushed back: the ticket names the role PowerUserPB-<stack>-exec, and adding the region overruns IAM's 64 characters for long region names; the README now says to name a second stack.
  8. A missing unshare passed the "namespaces are refused" check. Fixed: the script checks the program exists first.
  9. In the Dockerfile, a pipe hid cargo's exit status. Fixed: cargo writes to a file. The part about repeating filter-sandbox-container.sh's helper: Pushed back: a Docker build stage cannot call a script of the host.
  10. The README said Fargate "stays on" the tested platforms, where it is not yet listed. Fixed. The part about docs/develop.md linking the README on main: Pushed back: the guide's other links to files of this kind point at main too, and the link resolves once this merges.

Standards, round 1

  1. "Test seam" is not the project's language. Fixed: "the containment test and the runtime test" throughout.
  2. Two stack outputs nothing reads. Fixed: removed.
  3. The architecture mapping repeats, and one case had no arm for an unknown architecture. Fixed: the arm is added. The repetition: Pushed back: CloudFormation has no loop, and Dockerfile stages share no shell.

Spec, round 1

  1. The README's "stays on the tested platforms" runs ahead of spec Run grant workers in a filter sandbox on hosts without namespaces #21, where the reporter's run gates the listing. Fixed, with finding 10.

Security, round 1

No findings: backdating changes only notBefore, each leaf's key is made at issuance, and notAfter and the grant deadline are untouched.

Correctness, round 2

  1. When run-task fails for the second architecture, the first task is left running unwatched. Pushed back: that task ends by itself within minutes and its log stays in the log group, and the maintainer reruns once the failure is fixed.

Standards, Spec and Security, round 2

No findings.

Tracking

Epic: E1 Run grant workers in a filter sandbox on hosts without namespaces
Spec: #21 Run grant workers in a filter sandbox on hosts without namespaces
Ticket: #24 Qualify the filter sandbox on Fargate with a maintained harness
Flag: none

🤖 Generated with Claude Code

sodre and others added 5 commits November 20, 2026 00:57
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…#24)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The Gateway's certificates started at the second they were issued. On
Fargate 1.4.0 a client in the same task read a clock up to 10 ms behind and
rejected a leaf issued just after a second began as not yet valid.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…arness (#24)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… why a run failed (#24)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@sodre
sodre marked this pull request as ready for review November 20, 2026 06:42
@sodre
sodre merged commit 747ee18 into main Nov 20, 2026
38 checks passed
@sodre sodre mentioned this pull request Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Qualify the filter sandbox on Fargate with a maintained harness

1 participant