Repository navigation
fix(gateway): A client whose clock runs behind accepts a certificate the Gateway just issued - #36
Merged
Merged
Conversation
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…#24) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The Gateway's certificates started at the second they were issued. On Fargate 1.4.0 a client in the same task read a clock up to 10 ms behind and rejected a leaf issued just after a second began as not yet valid. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…arness (#24) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… why a run failed (#24) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
sodre
marked this pull request as ready for review
November 20, 2026 06:42
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #24
Summary
A client whose clock runs slightly behind the Gateway's now accepts a certificate the Gateway has just issued: the sidecar CA and every leaf start their validity one minute before they are issued, and a leaf never starts before the CA that signs it. Maintainers also get a harness, in
qualification/fargate, that runs the Gateway's containment test and its runtime test on real ECS Fargate 1.4.0, on x86-64 and arm64.The harness found the certificate bug on its first run, and the owner chose to widen this ticket and fix it here, although spec #21 lists changes to the issuer as out of scope. On Fargate's kernel,
time()advances once per 10 ms tick. The Gateway dated a leaf at the current second, so curl in the same task, readingtime(), rejected a leaf issued within 10 ms after a second began as not yet valid. The runtime test hit that at the same step on every run, on both architectures.Choices the ticket left open:
AWS_REGIONchooses, and the README says so.python:3.12-bookworm, which already has Python and curl. The two other needs (thecryptographywheel andlibnss_wrapper.so) are fetched for the target architecture and unpacked, so neither image needs emulation.run.sh runfails a task whose image was built from another commit than the checkout.Evidence
qualification/fargate/run.sh run(the final run, below), which deployed, built and ran from those filesqualification/fargate/run.sh run: 25 passed, 0 failed"filter", with the test grant authority and mTLS site inside the taskqualification/fargate/run.sh run: sidecar and shared mTLS passed, filter-killed grant worker fenced, exit code 0qualification/fargate/run.sh run: 23 passed, 0 failedqualification/fargate/run.sh run: sidecar and shared mTLS passed, filter-killed grant worker fenced, exit code 0gh issue comment 24: the comment of 2026-11-20 holds the table and both task logscountersign-isolation-probe-18(us-east-1) is deleted once the harness replaces itaws cloudformation delete-stackandwait stack-delete-complete, exit 0;describe-stacksthen answered "does not exist"AssertionErrorat the runtime test's last phase, andrun.sh runexited 1tests/sidecar.rs)Validity { not_before: 2026-11-20 6:18:07 ...}, the second of issuancelocal_shared_ca_checks_key_and_issues_for_tls)test result: FAILED. 0 passed; 1 failedshared_issuer.rs)left: 1795156239, right: 1795156294run.sh runrefuses a task built from another revisionaf961f2, checkout at30bedc2:did not run this checkout (v0.2.0-10-g30bedc2): build its image first, exit 1qualify.shrefuses an image withoutunshareunshareoff thePATH:The qualification image has no unshare., exit 1The final run,
qualification/fargate/run.sh runat30bedc2in us-east-2, exited 0. Both tasks: Fargate Linux 1.4.0, kernel 6.1.190, user namespaces refused, no bubblewrap, startup lineGateway worker sandbox "filter": non-dumpable, Landlock ABI v2, seccomp allow-list, exit code 0. Two earlier runs ataf961f2passed the same way.The cause was measured on Fargate with an instrumented copy of the runtime test: curl started at second .984, the leaf's
notBeforewas the next second, the coarse clock read about 10 ms behind the real one, and curl exited 35.Also run, all exit 0:
cargo test --locked -p countersign-gateway(lib 4 passed, bin 5 passed,sidecar6 passed,upstream1 passed, 0 failed, 0 ignored),cargo clippy -p countersign-gateway --all-targets -- -D warnings,scripts/filter-sandbox-container.sh(CI's container job, every step),runtime.pyunder the namespace sandbox on the host, andqualification/fargate/run.sh local.Not run: the comment on #18, since Gateway images are published only for a release tag, so the build to link exists after the owner's next release. The rest of the workspace's suites (pnpm, the other crates), which CI runs: the change touches only the Gateway crate.
Merge Danger
Door: two-way. A revert restores certificates that start at the second of issuance; nothing stored changes.
Blast Radius: every certificate the Gateway issues itself.
The next Gateway release carries the issuer change: the sidecar CA and each leaf from the sidecar CA or a local shared CA are valid from one minute before issuance. Leaves from AWS Private CA are unchanged. The merge deploys nothing. The stack
countersign-fargate-qualificationwas deleted from us-east-2 after the final run (run.sh teardown);run.shdeploys it again for the next qualification. No flag.Reviews
Correctness, Standards, Spec and Security, two rounds: 15 findings, each with its outcome
Round 1 pinned
9a0cb18, round 230bedc2.Correctness, round 1
aws ecs wait tasks-stoppedgives up after ten minutes, which a hung task outlasts, and the script then died before printing its log. Fixed: it waits twice and prints the log and exit code either way.run-taskexits 0 when Fargate places no task, and the reason was discarded. Fixed: the script prints the failure's reason and stops.local -Aneeds bash 4, which a stock Mac lacks. Fixed: an indexed array.get-log-eventsreturns one page, and its failure was swallowed. Fixed:filter-log-events, which follows the pages, and a named line when the task left no log.runwithoutbuildcould pass on an image from another commit. Fixed: a task has to name the checkout's revision.PowerUserPB-<stack>-exec, and adding the region overruns IAM's 64 characters for long region names; the README now says to name a second stack.unsharepassed the "namespaces are refused" check. Fixed: the script checks the program exists first.filter-sandbox-container.sh's helper: Pushed back: a Docker build stage cannot call a script of the host.docs/develop.mdlinking the README onmain: Pushed back: the guide's other links to files of this kind point atmaintoo, and the link resolves once this merges.Standards, round 1
casehad no arm for an unknown architecture. Fixed: the arm is added. The repetition: Pushed back: CloudFormation has no loop, and Dockerfile stages share no shell.Spec, round 1
Security, round 1
No findings: backdating changes only
notBefore, each leaf's key is made at issuance, andnotAfterand the grant deadline are untouched.Correctness, round 2
run-taskfails for the second architecture, the first task is left running unwatched. Pushed back: that task ends by itself within minutes and its log stays in the log group, and the maintainer reruns once the failure is fixed.Standards, Spec and Security, round 2
No findings.
Tracking
Epic: E1 Run grant workers in a filter sandbox on hosts without namespaces
Spec: #21 Run grant workers in a filter sandbox on hosts without namespaces
Ticket: #24 Qualify the filter sandbox on Fargate with a maintained harness
Flag: none
🤖 Generated with Claude Code