Skip to content

Gateway v0.2.0 cannot start a grant worker on ECS Fargate 1.4.0 #18

Closed

Description

@sodre

The v0.2.0 Gateway exits during its worker startup check on Amazon ECS Fargate Linux/x86-64, platform 1.4.0. This blocks deployment of a desktop-approved mTLS integration that passes end to end in Docker Compose with the same Gateway release.

Observed on Fargate

A standalone private task ran the published linux/amd64 manifest ghcr.io/zeroae/countersign-gateway@sha256:b4f55bb9020be6f34ebf3e29ef0d542ca6d24567ddb692b8e1fbe50e49b051fe using the image's default user and Fargate's default security settings. ECS reported exit code 1. The worker error was:

Gateway worker sandbox failed to start (exit status: 1): bwrap: No permissions to create new namespace

The supervisor invokes bwrap --unshare-all and starts a real worker before loading destination policy or contacting a Broker. Fargate does not expose the Docker security options used by the working Compose deployment. Released worker launcher · Gateway startup order · AWS task-definition limits

Reproduce without credentials

Run the image above in a Linux/x86-64 Fargate 1.4.0 task. Save the following as /tmp/probe.json inside the container.

{
  "listen": "127.0.0.1:8444",
  "proxy_url": "https://localhost:8444",
  "broker_url": "https://localhost:9443",
  "broker_ca": "/unused",
  "ca_output": "/unused",
  "shared": {
    "listener_cert": "/unused", "listener_key": "/unused", "gateway_token": "/unused",
    "ca": {"type": "local", "cert": "/unused", "key": "/unused"}
  },
  "destinations": [{
    "id": "probe", "version": 1, "pattern": "*.example.invalid:443",
    "private_networks": ["10.0.0.1/32"], "roots": "/unused"
  }]
}
  1. Run countersign-gateway env /tmp/probe.json. It exits successfully.
  2. Run countersign-gateway serve /tmp/probe.json. On Fargate it exits with the bubblewrap error above. On a host where the worker starts, the deliberately invalid destination produces private egress requires an exact host and prefix rule at the next startup step. This makes the result observable without creating a grant or supplying keys.

Requested alternative

Please provide a supported worker isolation option for deployments that cannot create nested Linux namespaces. One candidate is a restricted local worker that uses the existing Unix socket handoff. A separately isolated Fargate task is another candidate, but it would need an authenticated transport because the current handoff passes file descriptors. The existing startup check should fail closed when the selected isolation mechanism is unavailable. Worker handoff

We can validate a candidate against two concurrent grants on real Fargate. Each worker should have access only to its own streams, be unable to read supervisor credentials or open independent outbound connections, and stop serving on revocation. The positive end-to-end test is a paired Companion approving a client certificate, followed by the mock MCP tool reading the mTLS-protected flag into chat. Per-grant isolation contract

Activity

  1. added
    origin:siteOpened from a deployment site
    and removed on Nov 19, 2026
  2. sodre commented on Nov 19, 2026

    @sodre
    MemberAuthor

    This was generated by AI during triage.

    Brief (ready for human)

    Category: enhancement
    Summary: Give the Gateway a supported grant-worker isolation mechanism for hosts that cannot create Linux namespaces, such as ECS Fargate

    Why this needs a human:

    • Security design decision. The per-grant isolation contract in the design record requires each grant worker to be isolated "by user principal/container or equivalent OS sandbox". Someone must decide what counts as equivalent without namespaces, and record it as an ADR. Options include:
      • a worker running as the same user, with a seccomp filter that denies open/socket/ptrace once the worker has its streams;
      • a worker running as a separate user, which needs a root-started supervisor and contradicts the operator guide's "keep the supervisor unprivileged";
      • Landlock, depending on the Fargate kernel version (not yet checked);
      • a separate Fargate task per grant, which needs an authenticated transport in place of today's file-descriptor handoff.
    • External qualification. The result has to be proven on real Fargate with two concurrent grants. The reporter offered to do this.

    Verified:
    The failure reproduces locally. The arm64 build of the v0.2.0 Gateway image, run as its default non-root user, behaves like this under Docker's default seccomp/AppArmor profile:

    • countersign-gateway env probe.json exits 0.
    • countersign-gateway serve probe.json exits 1 with Gateway worker sandbox failed to start (exit status: 1): bwrap: No permissions to create new namespace.
    • With seccomp=unconfined and apparmor=unconfined (the Compose demo's exceptions), the same command reaches the next startup step, private egress requires an exact host and prefix rule, as the report predicts.

    Fargate does not let a task change its seccomp or AppArmor profile, so it cannot apply those exceptions.

    Current behavior:
    The supervisor's startup check launches a real bubblewrap worker (bwrap --unshare-all: new user, mount, PID and network namespaces). It fails closed when the host refuses namespaces. Bubblewrap is the only worker sandbox. The design record and the operator guide both state that the Gateway requires Linux with bubblewrap and unprivileged user namespaces, so v0.2.0 behaves as documented. This is a request for a second supported platform, not a defect.

    Desired behavior:
    An operator can select a worker isolation mechanism that works on a host without nested namespaces, such as Fargate platform 1.4.0. Every per-grant isolation property the contract states must hold there. Startup still fails closed when the selected mechanism is unavailable.

    Key interfaces:

    • The startup environment check, which probes a real worker before loading destination policy or contacting a Broker. It must probe whichever mechanism is selected.
    • The worker handoff, which passes three connected Unix stream endpoints per connection by SCM_RIGHTS. A local mechanism can keep it; a cross-task mechanism cannot.
    • Gateway configuration: a new isolation selector, with bubblewrap as the default, so existing deployments are unchanged.

    Acceptance criteria:

    • An ADR records which mechanism counts as an equivalent OS sandbox under the per-grant isolation contract, and why
    • With the new mechanism selected, a grant worker cannot read the supervisor's credentials or environment, cannot open an outbound connection of its own, and cannot reach another grant's streams or memory. Tests prove each of these with two concurrent grants.
    • Revocation stops a worker serving under the new mechanism
    • With the new mechanism selected but unavailable, startup fails closed with an error that names the mechanism
    • The default stays bubblewrap, and existing deployments behave exactly as before
    • The operator guide and the tested-platforms page describe the new mechanism. Fargate is listed as qualified only after a real-Fargate run, in which a paired Companion approves a client certificate and the mock MCP tool reads the mTLS-protected flag.

    Out of scope:

    • Relaxing or removing the fail-closed startup check
    • Changing the sidecar CA, issuer or destination-policy behavior
    • Other container platforms, unless the chosen mechanism covers them for free
  3. added theissue type on Nov 19, 2026
  4. sodre commented on Nov 20, 2026

    @sodre
    MemberAuthor

    This was generated by AI during triage.

    Update: design settled, spec in #21

    The security decision this issue was waiting on is made. The Gateway will get a second worker sandbox for hosts that refuse namespaces, such as Fargate. The work is planned in #21, the first epic (milestone E1).

    What's coming:

    • A new Gateway field, worker_sandbox: "namespaces" (the default, today's bubblewrap sandbox) or "filter". The Gateway never falls back from one to the other on its own, and startup still fails closed.
    • Every grant worker, in either sandbox, gets the same base layers: a seccomp allow-list that kills the worker on any forbidden call, Landlock as far as the kernel offers it, non-dumpable memory, and no inherited descriptors. A grant worker can signal only itself. A worker killed by the filter fences its grant, so the grant needs fresh approval.
    • Shared mode is supported under "filter".

    Evidence, from probes run on real Fargate 1.4.0 (kernel 6.1.190, x86-64 and arm64, on both the v0.2.0 Gateway image and Amazon Linux 2023):

    • bubblewrap is refused, exactly as you reported.
    • A seccomp filter installed by the task itself blocks file access, the supervisor's environment and memory, other workers' memory, TCP, UDP and running programs. The SCM_RIGHTS stream handoff keeps working.
    • Landlock is only ABI v2 there, so it covers files and running programs but not the network. It's a backstop, not the main mechanism.
    • Fargate can't add CAP_SYS_ADMIN, run privileged or use a custom seccomp profile, and AWS has said it has no plans to change that ([Fargate] [request]: Allow privileged mode aws/containers-roadmap#1000). So bubblewrap can't work on Fargate.

    ADR 0001 · Probe and Fargate results · Fargate capability research

    Taking up your offer: once #21 ships, could you qualify worker_sandbox: "filter" on your Fargate deployment? The run would use two concurrent grants: a paired Companion approves a client certificate, and the mock MCP tool reads the mTLS-protected flag into chat. Fargate goes on the tested-platforms page only after that run. We'll post here when there's a build to try.

  5. sodre commented on Nov 21, 2026

    @sodre
    MemberAuthor

    Reporter qualification passed on AWS ECS Fargate 1.4.0, Linux x86-64, on November 21, 2026. The shared Gateway used worker_sandbox: "filter". The Broker, Gateway and mTLS flag site ran in a standalone Fargate task. FINsight chat, its mock backend and MCP server ran in Docker Compose. The installed Linux ARM64 Companion v0.2.1 used a disposable PKCS#11 client identity on the desktop. Countersign v0.2.1

    1. The flag site rejected a direct TLS request without a client certificate.
    2. After pairing the Companion with FINsight, I submitted COUNTERSIGN-FLAG in two separate investigations. The Companion presented a distinct grant for each, and both approvals used the demo identity.
    3. The site held the first authenticated GET /flag open until the second authenticated request arrived. Each received HTTP 200. Both completed assistant messages displayed the site's random flag. The browser verifier hashed each rendered reply and matched it against the digest computed independently at the site.
    Evidence Observed value
    Release commit 7199c789e2e514cb6201ba84eed0071d9a0b50e0
    Published x86-64 Gateway image digest sha256:d070f8c64fae193f52cd572faf99e3361662daa5b14a2fefb34fb657e7b171d0
    Approved investigation contexts cs-qual-9ea18ed7204e4a and cs-qual-e1194b6f15976c
    SHA-256 of the site flag and each chat reply c795d976548633371a55eeaa694aa857ac3f045d4411bfab3121f3f88abf2e4c

    The Gateway logged Gateway worker sandbox "filter": non-dumpable, Landlock ABI v2, seccomp allow-list. The site recorded two TLS 1.3 connections from distinct source ports, 44278 and 44280, using the same client certificate. They reached the barrier 1.398 seconds apart. The qualification run completed and tore down the temporary resources. FINsight qualification run and CloudWatch excerpts

  6. sodre commented on Nov 22, 2026

    @sodre
    MemberAuthor

    The filter sandbox shipped in v0.2.1: set worker_sandbox: "filter" on hosts that refuse namespaces, such as ECS Fargate. With the qualification run above, ECS Fargate 1.4.0 x86-64 is now listed as qualified on the handbook's Tested platforms page (#47). Fargate arm64 has passed the maintainers' harness, but no end-to-end run has been made on it yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions