Skip to content

feat(gateway): An operator can run the Gateway under the filter sandbox where namespaces are refused - #34

Merged
sodre merged 4 commits into
mainfrom
feat/23-worker-sandbox-filter
Nov 20, 2026
Merged

sodre merged 4 commits into
mainfrom
feat/23-worker-sandbox-filter

Conversation

@sodre

@sodre sodre commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Closes #23

Summary

An operator on a host that refuses namespaces, such as ECS Fargate, can set worker_sandbox to "filter" in the Gateway configuration, and the Gateway starts and serves grants there without bubblewrap. With the field left out nothing changes: the namespace sandbox stays the default, the Gateway never switches from one sandbox to the other, and a sandbox that cannot start stops the Gateway with worker sandbox "<sandbox>" failed: <layer>: <cause>.

countersign-gateway serve gateway.json
  Config::load                      # worker_sandbox: "namespaces" (default) | "filter"; anything else fails
  ChildHandle::verify_environment   # probes a real grant worker under the chosen sandbox
    launch_command
      "namespaces" -> bwrap ... /worker grant-worker     # as before
      "filter"     -> <the Gateway executable> grant-worker
                        cleared environment
                        parent-death signal               # what --die-with-parent gave
                        every descriptor above 2 close-on-exec
    grant worker: confine()         # the base layers, the same under both
  on failure: exit 1
    worker sandbox "<sandbox>" failed: <layer>: <cause>
    See "Choose a worker sandbox" in the operator guide: <link>

<layer> is non-dumpable, Landlock or seccomp allow-list when a base layer failed, namespaces when bubblewrap or its namespaces did, and launch when the filter sandbox could not start the grant worker at all.

Choices the ticket left open:

  • A grant worker does not outlive its supervisor under "filter". bubblewrap's --die-with-parent gave this under the namespace sandbox. Started directly, a grant worker would wait forever on its control socket after the supervisor was killed, holding a grant's traffic keys. The launch sets a parent-death signal, and the runtime test now asserts it under both sandboxes.
  • How the failed layer reaches the supervisor. A grant worker that cannot apply a base layer exits with status 64, 65 or 66 (non-dumpable, Landlock, allow-list) and writes the cause to standard error. bubblewrap passes the status on. Reading the layer out of the error text broke as soon as a warning came first.
  • No kernel floor beyond the base layers'. Marking every descriptor close-on-exec in one call needs Linux 5.11. Where the kernel or an older container profile refuses that call, the launch lists /proc/self/fd itself between fork and exec and marks each one, and fails when it cannot. Spec story 9 asks that older kernels are not locked out.
  • launch as a layer name. ADR 0001's layers cannot name a failure to start the process, and the error's form needs one. The operator guide defines it where the operator reads it.
  • The CI job is a script, scripts/filter-sandbox-container.sh, so a maintainer can run the same thing locally. It builds on the host and runs in ubuntu:24.04 as an unprivileged user. Two images: one without bubblewrap (the filter sandbox does not need it), one with (the namespace sandbox fails there because the host refuses namespaces, not because a program is missing). Two last steps leave Docker's default profile for one quirk each: a host that refuses the allow-list, and a kernel without close_range.
  • The job runs on x86-64 and arm64, because the allow-list is written per architecture.

Extras for the owner

  • The runtime test's cutoff assertion after Bob's retirement fails intermittently, on main too. assert site.cutoff.wait(1) in the shared-mode retirement part of crates/countersign-gateway/tests/runtime.py failed 2 of 6 runs of main's test against main's binary on this arm64 host, and at a similar rate on this branch under both sandboxes. It predates this ticket and nothing here touches it. No source settles it.

Evidence

Criterion Ticked Shown by
The configuration accepts optional worker_sandbox, "namespaces" (default) and "filter"; any other value fails loading yes The configuration loads only the two values
Under "filter", grant workers start without bubblewrap and without /usr/bin/bwrap, in sidecar and shared mode yes Filter grant workers are started by the Gateway itself; scripts/filter-sandbox-container.sh step 3, in an image with no bubblewrap
There is no automatic fallback in either direction yes The namespace sandbox refuses to start where namespaces are refused; The filter sandbox refuses to start where the allow-list is refused
A startup failure exits non-zero, names the sandbox and the failed layer, points to the operator guide, and does not suggest the other sandbox yes The same two tests, which read the exit status and both lines of the error
A new CI job, in a container under Docker's default profile, runs the runtime test under "filter" (sidecar, shared, concurrent users, cutoff, revocation) yes The job gateway-filter-sandbox on 36d2eda, x86-64 and arm64 runners (https://github.com/zeroae/countersign/actions/runs/44679962288): sidecar Gateway mTLS passed, shared Gateway mTLS passed, filter-killed grant worker fenced under Gateway worker sandbox "filter"
The same job runs the worker sandbox containment test yes The same job: 25 passed, 0 failed on x86-64, 23 passed, 0 failed on arm64
The same job proves that "namespaces" fails at startup there with the named error yes The same job: worker sandbox "namespaces" failed: namespaces: bwrap: No permissions to create new namespace, ..., sidecar and shared
The configuration reference documents worker_sandbox and both values yes mkdocs build --strict, which resolves the row's link to the new section
The operator guide has a section on choosing a worker sandbox yes mkdocs build --strict; the startup error's link ends in the section's anchor, operate/#choose-a-worker-sandbox, which the startup tests assert
The troubleshooting row covers both sandboxes yes mkdocs build --strict, which resolves the row's link
The design record's platform line names both worker sandboxes yes git diff origin/main...HEAD -- docs/specs/2026-11-17-countersign-design.md: one line, naming the namespace sandbox and the filter sandbox
Test Red Green
The configuration loads only the two values (runtime.py, configuration_loading) Before the field existed: AssertionError: namespaces ok
Filter grant workers are started by the Gateway itself (runtime.py, launched_directly) With "filter" routed through bubblewrap: AssertionError: Name: bwrap ok
The namespace sandbox refuses to start where namespaces are refused (--worker-sandbox namespaces --startup-fails namespaces, script step 4) With try "filter" added to the message: AssertionError; with "namespaces" routed to the direct launch, a fallback: the Gateway started, TimeoutExpired ok
The filter sandbox refuses to start where the allow-list is refused (--worker-sandbox filter --startup-fails 'seccomp allow-list', script step 5, image with bubblewrap) Under the default profile, where the filter sandbox works: the Gateway started, TimeoutExpired, exit 1. With the exit-status mapping shifted by one: worker sandbox "filter" failed: Landlock: ..., AssertionError ok
No grant worker outlives its supervisor (runtime.py, after the filter-kill scenario) Without the parent-death signal: AssertionError: A grant worker outlived its supervisor ok
The filter sandbox's launch passes on no descriptor and no environment (unit test filter_sandbox_launch_passes_on_no_descriptor_and_no_environment) With close_range starting at 1,000,000: FAILED; with env_clear removed: FAILED ok
The same test where close_range is refused (script step 6) With the listing marking nothing: FAILED, exit 101 ok

What else ran, on an arm64 host (Linux 6.17, Docker 29.2.1, bubblewrap 0.9.0), at 36d2eda:

  • scripts/filter-sandbox-container.sh: exit 0, six steps. The container refuses namespaces (bwrap: No permissions to create new namespace). Containment test: 23 passed, 0 failed. Runtime test under "filter": sidecar Gateway mTLS passed, shared Gateway mTLS passed, filter-killed grant worker fenced, with the log Gateway worker sandbox "filter": non-dumpable, Landlock ABI v7, seccomp allow-list. Step 4 printed worker sandbox "namespaces" failed: namespaces: bwrap: No permissions to create new namespace, ...; step 5 printed worker sandbox "filter" failed: seccomp allow-list: Invalid argument (os error 22); each followed by See "Choose a worker sandbox" in the operator guide: https://zeroae.github.io/countersign/operate/#choose-a-worker-sandbox.

  • runtime.py target/debug/countersign-gateway, with no option and with --worker-sandbox filter, on the host: all three lines each. Each needed retries for the flake under Extras.

  • cargo test --locked -p countersign-gateway: 14 unit and integration tests passed and the containment test's 23, 0 ignored, 0 failed.

  • cargo fmt --all -- --check, cargo clippy --locked -p countersign-gateway --all-targets -- -D warnings, mkdocs build --strict: clean.

  • Probe (2026-11-19): on main, runtime.py with main's binary, six runs: four passed, two failed at assert site.cutoff.wait(1).

  • GitHub, on 36d2eda: all 28 checks passed. The job gateway-filter-sandbox ran all six steps of the script on x86-64 and arm64 runners, and the workspace jobs ran cargo test --workspace, pnpm lint and pnpm test.

Not run: nothing

Merge Danger

Door: two-way. A revert removes the field; a configuration that sets worker_sandbox then fails to load as an unknown field, so an operator who set it removes it first. No release carries the field yet.

Blast Radius: Gateway startup and grant worker launch.

The merge ships no image and no infrastructure. It adds one CI job to the end-to-end workflow. For a Gateway that leaves worker_sandbox out, two things change at the next release: a failed startup probe prints worker sandbox "namespaces" failed: namespaces: <cause> in place of Gateway worker sandbox failed to start (...), and a grant worker that cannot apply a base layer exits with 64, 65 or 66 in place of 1. No flag.

Reviews

Correctness, Standards, Spec and Security, three rounds: 24 findings, each with its outcome. Round 3 found nothing.

Pinned commits: round 1 7b85f451de2439a1ddf69de642118de9c93b9e80, round 2 11bed34dbeba058e81e5e6fef99615da8a59af6a, round 3 36d2edaa9166242fa5142f244f0b08b93b3f6a33. Security found nothing in any round.

Correctness, round 1

  1. The failed layer was read out of the probe's standard error by its prefix, so a warning ahead of it blamed the wrong layer. Fixed: the grant worker's exit status carries the layer.
  2. The direct launch has no counterpart to bubblewrap's --cap-drop ALL, --new-session and --chdir /. Pushed back: the allow-list leaves a confined grant worker no system call that a capability, a session or a working directory changes (signals only to itself, no file or socket call, ioctl only for FIONBIO), it confines itself before reading any input, and the operator guide keeps the supervisor unprivileged.
  3. A failed launch reached the operator as a bare error number, hiding the Linux 5.11 requirement. Fixed: the requirement is gone; the launch marks descriptors one by one where the kernel refuses the single call.
  4. The test's check that the other sandbox is not named matched the bare word, which the allow-list's own failure text contains. Fixed: it looks for the quoted value, and the text says "the allow-list killed".
  5. The script assumed cargo's target directory is ./target. Fixed: it takes each executable's path and the target directory from cargo, and mounts the directory.
  6. The filter-fails step depended on the host kernel offering Landlock. Fixed: the step refuses the allow-list's own call, which every kernel reaches.
  7. The new unit test depended on HOME being set and standard input being open. Fixed: it duplicates a descriptor it opened, and reads the launched environment with env.
  8. Option parsing unpacked before the assertion that prints the usage. Fixed.
  9. The sandbox's names lived in several hand-kept places. Fixed: WorkerSandbox::outer_layer serves the startup log and the error.

Standards, round 1

  1. The developer guide's new paragraph split the SIGSYS section's text. Fixed: it moved under "Run the checks".
  2. A test comment called grant workers "its child", which the glossary avoids. Fixed.
  3. launch is a layer that neither ADR 0001 nor the glossary defines. Pushed back: the error's form needs a layer for a grant worker that could not be started at all, the ticket itself says the supervisor "launches each grant worker directly", and the operator guide defines the name beside the others.
  4. Possible Primitive Obsession: the probe's failure was a string re-parsed for its layer. Fixed: it is a layer and a cause.
  5. Possible Repeated Switches on the sandbox for the outer layer's name. Fixed, with 9.
  6. Possible Mysterious Name: sandbox(sandbox, ...) and direct. Fixed: launch_command, namespace_sandbox, filter_sandbox.
  7. Possible Duplicated Code: the script's last docker run repeated inside. Fixed: inside takes the profile.
  8. Possible Duplicated Code in configuration_loading. Fixed.
  9. Possible Divergent Change: do_test takes a third flag. Pushed back: the function held several scenarios behind flags before this ticket, and splitting the runtime test is not this ticket's work.

Spec, round 1

  1. The direct launch needed Linux 5.11, which neither the ticket nor the spec decided, against spec story 9. Fixed, with 3.
  2. Filter-sandbox grant workers are killed when the supervisor dies, which no criterion asks for. Pushed back: it keeps what --die-with-parent gave under the namespace sandbox; without it a grant worker holding traffic keys outlives a killed supervisor. Named in the Summary as a choice the ticket left open.

Correctness, round 2

  1. The script left its first temporary profile behind. Fixed: one file, removed on exit.

Standards, round 2

  1. A new comment called the grant worker bubblewrap's "child". Fixed.
  2. The developer guide and the script's header still described two tests under the default profile. Fixed: both say what the two last steps do.

Spec, round 2

  1. The fallback marked only descriptors listed before the fork, dropped fcntl's result, and said nothing. Fixed: the listing runs between fork and exec by system calls alone, and the launch fails when it cannot list or mark a descriptor.

Tracking

Epic: E1 Run grant workers in a filter sandbox on hosts without namespaces
Spec: #21 Run grant workers in a filter sandbox on hosts without namespaces
Ticket: #23 Operators can choose the filter sandbox
Flag: none

🤖 Generated with Claude Code

sodre and others added 4 commits November 19, 2026 23:55
The Gateway configuration takes an optional `worker_sandbox`: "namespaces"
(the default) or "filter". Under "filter" the supervisor starts each grant
worker itself, with no bubblewrap: a cleared environment, no descriptor
beyond the control socket and the standard streams, and a parent-death
signal in place of bubblewrap's --die-with-parent. The grant worker then
confines itself with the base layers, as under the namespace sandbox. The
Gateway never falls back from one sandbox to the other.

When the chosen sandbox cannot start, startup exits with
`worker sandbox "<sandbox>" failed: <layer>: <cause>` and a pointer to the
operator guide's new section on choosing a worker sandbox.

A new job, gateway-filter-sandbox, runs the containment test and the
runtime test under "filter" in a container under Docker's default security
profile, on x86-64 and arm64, and proves that "namespaces" fails at startup
there with the named error.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ap at hand (#23)

A container whose profile refuses one call of the Landlock layer: the
Gateway under "filter" exits with the named error and starts no grant
worker another way. The developer guide's line on the script moves to a
paragraph of its own.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…nels launch (#23)

- A grant worker that cannot apply a base layer exits with a status that
  names the layer, so the startup probe no longer reads it out of the
  standard error text, where a warning ahead of it hid it.
- The filter sandbox's launch no longer needs Linux 5.11: where the kernel
  or the container profile refuses close_range, the descriptors listed
  before the fork are marked close-on-exec one by one. The docs drop the
  kernel floor.
- The container script takes the executables' paths from cargo, mounts
  cargo's target directory, and gains two steps: the filter sandbox fails
  by name where the host refuses the allow-list, and passes on no
  descriptor where close_range is refused.
- The runtime test's --startup-fails takes the layer it expects, and looks
  for the other sandbox's quoted value, not the bare word.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…23)

Where close_range is refused, the grant worker's launch now lists the open
descriptors itself, between fork and exec, by system calls alone, and
fails when it cannot list or mark one. The listing no longer happens in
the supervisor ahead of the fork, where a descriptor opened in between
would have been missed.

The container script removes its temporary profile, and it and the
developer guide say what its two last steps do.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@sodre
sodre marked this pull request as ready for review November 20, 2026 05:36
@sodre
sodre merged commit f13433d into main Nov 20, 2026
38 checks passed
@sodre sodre mentioned this pull request Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Operators can choose the filter sandbox

1 participant