Skip to content

feat(gateway): Every grant worker confines itself before it reads any input - #26

Merged
sodre merged 8 commits into
mainfrom
docs/18-worker-sandbox
Nov 20, 2026
Merged

sodre merged 8 commits into
mainfrom
docs/18-worker-sandbox

Conversation

@sodre

@sodre sodre commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Closes #22

Summary

Every grant worker now locks itself down before it reads anything: its memory cannot be read by another process, Landlock gives it no file or network access where the kernel has Landlock, and the kernel kills it on any system call a grant worker does not need. The Gateway logs these layers at startup, and when the kernel kills a grant worker this way the log says so and that grant needs fresh approval.

countersign-gateway grant-worker            (inside bubblewrap, as before)
  worker_sandbox::confine()                 new, public: Layers or ConfineError{layer}
    non-dumpable
    no_new_privs
    Landlock: every right of the kernel's ABI handled, none granted (or "unavailable")
    seccomp allow-list: default kills the process
  worker::serve(layers)                     first read of the control socket

supervisor
  startup probe  -> "Gateway worker sandbox "namespaces": namespaces, non-dumpable, Landlock ABI v7, seccomp allow-list"
  worker ends    -> "Gateway grant <id>: grant worker killed by the worker sandbox's filter (SIGSYS); grant fenced"

Choices the ticket left open:

  • confine() takes no worker sandbox argument. The base layers are the same under both sandboxes, so the argument would do nothing yet. Operators can choose the filter sandbox #23 Operators can choose the filter sandbox adds the choice.
  • no_new_privs is set before Landlock, not after it. The ticket lists it with the allow-list, third. An unprivileged landlock_restrict_self fails with EPERM without it (landlock_restrict_self(2)), so it has to come second. A failure to set it is reported under the allow-list, the layer it belongs to.
  • Conditional rules. kill and tgkill only for the grant worker's own process ID. sendto only with a null destination, since Rust writes a stream with it. fcntl only for the three commands a grant worker uses, which keeps F_SETOWN from having the kernel signal another process. ioctl only for FIONBIO, socketpair only for AF_UNIX. clone only with CLONE_THREAD, for hosts whose profile answers clone3 with ENOSYS, as Docker's default does.
  • The startup probe reports from a thread with an async runtime, where a connection is served, so a host the allow-list cannot serve on fails at startup and not on the first grant.
  • A grant worker that ends on its own is fenced when it ends, whatever the cause, and one line is logged. Before, it was fenced when the next connection failed to attach.
  • tests/runtime.py is extended, not rewritten: it now asserts the startup log line and adds a third pass that delivers SIGSYS to a grant worker. Its existing assertions are as they were.
  • CI names the system call when a filter kill fails a run: a step that prints the kernel's audit lines, on failure only.
  • CI runs the Gateway on arm64 Linux: a gateway-arm64 job runs the Gateway crate's tests and the runtime test on ubuntu-24.04-arm. The owner added this criterion to Grant workers confine themselves under the namespace sandbox #22 after the session's retro.
  • docs/adr/ is left out of the handbook build, like docs/specs/ and docs/agents/. The strict build failed on ADR 0001, a page outside the navigation.

Left for #23 Operators can choose the filter sandbox: the startup error still reads Gateway worker sandbox failed to start (...). When the allow-list kills the startup probe, that message carries the signal and no layer name. #23 owns the error that names the sandbox and the layer.

The untracked folder crates/countersign-gateway/prototype-isolation-probe/ (a stray Cargo.lock) was in the worktree before this work and is not part of it.

From the session's retro, as the owner answered:

Extras for the owner

Evidence

Criterion Ticked Shown by
The worker sandbox module is public in the Gateway library, with one confine operation that reports the applied layers or names the layer that failed yes A confined grant worker reports its layers; A layer the host refuses fails confinement by name
Every grant worker, including the startup probe's worker, confines itself before reading any input yes The grant worker is confined before it is sent anything; runtime.py startup log
Startup logs the worker sandbox and each layer applied, including the Landlock ABI version or "Landlock unavailable" yes runtime.py startup log; Without Landlock it says so and is still confined
A new Rust integration test confines two concurrent stand-in grant workers ... yes Containment, two stand-ins; cargo test --workspace --locked
The allow-list rejects a foreign architecture, and the x32 range on x86-64 yes cargo test --workspace --locked in CI on x86-64
A grant worker killed by SIGSYS is logged as killed by the filter, and its grant is fenced and needs fresh approval yes runtime.py filter kill
The existing runtime test passes unchanged under the namespace sandbox in CI yes python crates/countersign-gateway/tests/runtime.py target/debug/countersign-gateway in CI
ADR 0001 and GLOSSARY.md are merged with this change yes git diff --stat origin/main...HEAD
CI runs the Gateway crate's tests, the worker sandbox containment test among them, and the runtime test on arm64 Linux yes The gateway-arm64 job, run 44662603958
A host that refuses Landlock's version query with EPERM is reported as Landlock unavailable, and the grant worker is still confined yes A host that denies Landlock is reported as without it and still confined
Test Red Green
Containment, two stand-ins (tests/worker_sandbox.rs: stream and threads, file, supervisor's environment, TCP, UDP, running a program, signalling another process, signalling itself, SIGSYS) With allow_list() not called: udp 54904: expected a refusal, got GotThrough; signal 0 9: expected a refusal, got Killed(9); trace-me: expected death by SIGSYS, got GotThrough ok
An unconfined process cannot read a grant worker's memory With PR_SET_DUMPABLE not set: left: GotThrough, right: Errno(1) ok
The grant worker is confined before it is sent anything With a read of the control socket placed before confine(): the grant worker waited for input unconfined ok
Without Landlock it says so and is still confined With ENOSYS no longer mapped to unavailable: the stand-in never served its stream ok
A layer the host refuses fails confinement by name With the Landlock error labelled as the allow-list: left: "seccomp allow-list: Operation not permitted (os error 1)" ok
A host that denies Landlock is reported as without it and still confined Before EPERM counted as unavailable: Landlock: Operation not permitted (os error 1), the stand-in never served its stream ok
Its threads work, and the grant worker starts, on a host without clone3 Without the clone rule: the probed grant worker ended with signal: 31 (SIGSYS) ok
Starting another process is refused Before the clone rule, on the host without clone3: the stand-in never served its stream ok
Having the kernel signal another process is refused With fcntl unrestricted: signal-on-io 723768: expected a refusal, got GotThrough ok
Returning from a signal handler works Without rt_sigreturn: left: Killed(31), right: GotThrough ok
runtime.py startup log With the startup eprintln! removed: AssertionError: [] ok
runtime.py filter kill (logged, fenced, Bob unaffected, a fresh approval served) With bubblewrap's exit code 159 not recognised: log has grant worker ended (exit status: 159), then AssertionError at the wait for the filter-kill line ok

Not shown red on its own: the x32 check (an allow-list kills an x32 number with or without the explicit range check), and the two memory-read checks with only the allow-list removed (non-dumpable memory still refuses them; the unconfined controls show the reads get through with no worker sandbox).

At commit 9fb3b25, the head, which adds the Landlock EPERM widening: cargo test --locked -p countersign-gateway locally on aarch64 gives worker_sandbox: 22 passed, 0 failed, and the same binary gives 22 of 22 in docker run ubuntu:24.04 under Docker's default profile. A fresh criteria judge read the widening's diff and called the criterion met, with no path that treats a Landlock failure after the version query as unavailable. No review round read this commit: three had run.

At commit 4f3ab9f: the gateway-arm64 job of run 44662603958 (ubuntu-24.04-arm) passed cargo test --locked -p countersign-gateway and the runtime test, and its x86-64 job passed too. At 69e7cff the same job printed worker_sandbox: 21 passed, 0 failed, then sidecar Gateway mTLS passed, shared Gateway mTLS passed, filter-killed grant worker fenced, so the runtime test does not flake on CI's arm64 as it does on the local machine. The Documentation workflow (mkdocs build --strict) failed at 69e7cff on ADR 0001 outside the navigation and passes at 4f3ab9f (run 44662603962). The new job was not shown red: a CI job has nothing to break but its own steps.

At commit e685181, whose Rust and Python files the head leaves unchanged:

  • cargo test --workspace --locked in CI (run 44655780164, ubuntu-latest, x86-64): pass. worker_sandbox: 23 passed, 0 failed, with a_system_call_in_the_x32_range_kills_it and a_system_call_from_a_foreign_architecture_kills_it both run, not skipped. Windows job: pass.
  • python crates/countersign-gateway/tests/runtime.py target/debug/countersign-gateway in CI (run 44655780170, ubuntu-24.04, x86-64, the real Gateway under bubblewrap): sidecar Gateway mTLS passed, shared Gateway mTLS passed, filter-killed grant worker fenced. Log: Gateway worker sandbox "namespaces": namespaces, non-dumpable, Landlock ABI v7, seccomp allow-list and Gateway grant alice: grant worker killed by the worker sandbox's filter (SIGSYS); grant fenced.
  • cargo test --locked -p countersign-gateway, local, aarch64, kernel 6.17: unit 3 and 4, sidecar 5, upstream 1, worker_sandbox 21 passed, 0 failed (the two x86-64 checks are compiled out). No skips.
  • The same worker_sandbox binary in docker run ubuntu:24.04 under Docker 29.2.1's default seccomp profile, aarch64: 21 passed, 0 failed. The first try there, on the build without the clone rule, was killed at clone (audit syscall=220), which is what round 1's finding 1 predicted.
  • The runtime test, local, aarch64: 1 of 4 runs passed. The other 3 failed at assert site.cutoff.wait(1) after Bob's retirement, which also fails 2 of 3 runs on main's code here (third extra above).
  • strace -D -f over the whole runtime test, local: exactly one killed by SIGSYS, the deliberate one.
  • Probe: bubblewrap 0.9.0 reports a child's fatal signal as exit code 128 plus the signal; bwrap ... sh -c 'kill -SYS $$' exits 159. Recorded beside killed_by_filter.
  • cargo +stable fmt --all -- --check and cargo +stable clippy --workspace --all-targets --locked -- -D warnings: clean.

Not run: Fargate (kernel 6.1, Landlock ABI v2), which is #24 Qualify the filter sandbox on Fargate with a maintained harness; the foreign-architecture check on arm64, where a 64-bit process cannot make a 32-bit call.

Merge Danger

Door: two-way. A revert removes the base layers; nothing stored changes.

Blast Radius: every grant worker, on every deployment.

The merge changes the Gateway executable and so its image; no configuration or infrastructure changes, and there is no flag. From this merge a system call a grant worker newly needs, from a dependency or C library update, kills grant workers until it is added to the allow-list. The startup probe, the runtime test and the containment test are what catch that first.

Reviews

Correctness, Standards, Spec and Security, three rounds (pinned c83a7e3, e685181, then 8b57275): 31 findings, each with its outcome

Correctness, round 1

  1. clone3 is allowed but not clone, so on a host that answers clone3 with ENOSYS the first thread kills the grant worker. Fixed: clone is allowed with CLONE_THREAD.
  2. The startup probe never exercises a thread or the async runtime. Fixed: it reports from a thread that builds the runtime.
  3. rt_sigreturn is missing, so returning from a signal handler is a filter kill. Fixed.
  4. Only ENOSYS and EOPNOTSUPP count as Landlock unavailable; EPERM from a container profile fails startup. Fixed: EPERM on the version query counts as unavailable.
  5. The worker-end log lines carry no identity. Fixed: they name the grant.
  6. The foreign-architecture, file and memory checks depend on the host. Fixed: no i386 entry point, Yama scope 2 and above, and /etc/hostname are each handled.
  7. A blocking recv hangs the test if the grant worker dies before reporting. Fixed: a read timeout.
  8. launch keeps an exited flag beside two channels for one fact. Fixed: the flag is gone.
  9. The one-byte report is encoded and decoded in two functions. Fixed: report and reported, side by side.
  10. The startup log hard-codes "namespaces". Pushed back: the worker sandbox choice and its type arrive with Operators can choose the filter sandbox #23 Operators can choose the filter sandbox.

Standards, round 1

  1. Same as 4. Fixed: as 4.
  2. The startup error names neither the sandbox nor the guide. Pushed back: that error is a criterion of Operators can choose the filter sandbox #23 Operators can choose the filter sandbox.
  3. New locals name the grant worker child, a word the glossary avoids. Fixed: process, in the lines this change adds.
  4. Layer::NoNewPrivs is not one of the ADR's base layers. Fixed: removed; the failure is reported under the allow-list.
  5. Same as 9. Fixed.
  6. Same as 10. Pushed back: as 10.
  7. start returns a bare tuple. Pushed back: a private function with two callers; the lifecycle state it sat beside is reduced by 8.
  8. Unclear names in the test (host, done, answer). Fixed: setup, checked, fail_with.
  9. Two drop guards with the same body in the test. Fixed: one Reaped.
  10. The CI step is copied into two workflows. Pushed back: two three-line steps; sharing them takes a composite action.

Spec, round 1

  1. The layer order differs from the ticket, with a fourth layer name. Fixed: three layer names; the order stays, since the kernel requires it, and the Summary says so.
  2. Every worker exit is now fenced and logged, not only SIGSYS. Pushed back: the watcher has to exist to see SIGSYS, a worker that ended was already fenced at its next connection, and the second line is what tells a crash from a filter kill.
  3. The CI steps were not asked for. Pushed back: they run on failure only and name the system call, which the spec's story 33 needs to be acted on.

Security, round 1

No finding at reportable confidence. Below it, and fixed here: fcntl was unrestricted, so F_SETOWN could have the kernel signal another process. Fixed: three commands only.

Correctness, round 2

No findings.

Standards, round 2

  1. When the allow-list kills the startup probe, the startup error names no layer. Pushed back: Operators can choose the filter sandbox #23 Operators can choose the filter sandbox owns that error; the Summary names this case for it.

Spec, round 2

  1. The exited guard in attach was removed. Pushed back: it answers finding 8; a send to a grant worker that ended fails and fences on the spot, and the watcher fences when it ends.
  2. clone3 stays unconditional, so a new process is refused only through clone. Carried over to The allow-list tells a thread from a new process under clone3 #28 The allow-list tells a thread from a new process under clone3

Security, round 2

No finding at reportable confidence. Below it: the clone3 point of finding 26, and that the clone rule would pass CLONE_THREAD with a namespace flag, which needs a capability no grant worker has.

Correctness, round 3 (the widening; Standards and Spec found nothing, and Spec judged the added criterion met)

  1. The failure step's audit lines mix the containment test's deliberate kills with a real one. Pushed back: each line carries its time and process, and the step runs only when the job failed.
  2. The job's comment said x86-64 runs both tests "below". Fixed: it says where each runs.
  3. The bubblewrap probe and the runtime step are copied from the x86-64 job. Pushed back: sharing them takes a composite action, for two jobs.
  4. The job installed the demo Integrator's whole fixture list. Fixed: cryptography alone.
  5. The runtime test relied on cargo test having built the Gateway. Fixed: an explicit build step.

The three round-3 fixes change the workflow only, in commit 4f3ab9f. No review round read them; the job's pass at that commit is their check.

Tracking

Epic: E1 Run grant workers in a filter sandbox on hosts without namespaces
Spec: #21 Run grant workers in a filter sandbox on hosts without namespaces
Ticket: #22 Grant workers confine themselves under the namespace sandbox
Flag: none

🤖 Generated with Claude Code

sodre and others added 4 commits November 19, 2026 19:31
ADR 0001 defines the equivalent OS sandbox the per-grant isolation
contract allows on hosts that cannot create namespaces, such as ECS
Fargate. GLOSSARY.md starts with the four terms it uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#22)

Every grant worker, the startup probe's included, now applies the base
layers of ADR 0001 to itself before it reads its control socket:
non-dumpable memory, Landlock with no rules granted, and a seccomp
allow-list whose default kills the process. Startup logs the worker
sandbox and each layer, and a grant worker that dies of SIGSYS is logged
as killed by the filter and its grant fenced.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…proval (#22)

The memory-read checks now opt out of Yama's descendant-only rule, so the
worker sandbox is what refuses them, and an unconfined reader checks
non-dumpable memory on its own. The Gateway's own grant worker is shown
confined before anything is sent to it. After a filter kill, a fresh
approval restores service.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…on needs (#22)

Review round 1. Docker's default profile answers clone3 with ENOSYS, so
the allow-list now lets clone start a thread and nothing else. fcntl is
down to the three commands a grant worker uses, which keeps F_SETOWN from
having the kernel signal another process. rt_sigreturn is allowed, so a
crash a handler sees first is not logged as a filter kill. The startup
probe reports from a thread with an async runtime, the worker-end log
lines name the grant, and a failed no_new_privs is reported under the
allow-list, the layer it belongs to.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
sodre and others added 4 commits November 19, 2026 21:32
The seccomp allow-list is written per architecture and CI ran only
x86-64. A new job runs the Gateway crate's tests, the containment test
among them, and the runtime test on ubuntu-24.04-arm.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The strict handbook build failed on docs/adr/0001, a page outside the
navigation. ADRs are maintainer records, read in the repository.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Round 3. It installs cryptography alone, builds the Gateway explicitly,
and its comment says where x86-64 runs each test.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A container profile older than Landlock answers its version query with
EPERM. ADR 0001 makes Landlock a backstop the Gateway starts without, so
that host is now reported as Landlock unavailable, as one with no
Landlock is. A failure after the kernel has named an ABI is still an error.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@sodre
sodre marked this pull request as ready for review November 20, 2026 03:00
@sodre
sodre merged commit ebbcba8 into main Nov 20, 2026
34 checks passed
@sodre sodre mentioned this pull request Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Grant workers confine themselves under the namespace sandbox

1 participant