Repository navigation
Conversation
…an egress proxy An optional top-level `egress_proxy` (an http URL with a host and a port) sends every destination through the operator's HTTP proxy, unless its rule says `"egress": "direct"`. On a proxied route the supervisor resolves only the proxy's host, on every connection and outside `denied_networks`, sends a plain-text CONNECT with the site's canonical name and port, and hands the tunnel to the egress relay where the direct socket went. It resolves nothing for the site and never tries a proxied destination directly. Until the failure values land, any failure of the proxied leg is today's 502 `upstream_failure`. Loading refuses an https proxy URL, one with credentials, any other `egress` value, and `private_networks` on a proxied rule, naming the rule and both fixes. `env` and `serve` refuse the same configurations, and print the cause in its own words. Startup logs the egress proxy and the rules that leave their address policy to it. The runtime test gains a test-only CONNECT proxy that records each request line. Through it, in sidecar and shared mode, the Workload reads the site with the approved certificate while the Gateway cannot resolve the site's name; one Gateway serves a proxied and a direct rule with `private_networks`; lease lapse and retirement cut a proxied connection; and proxy environment variables without `egress_proxy` change nothing. ADR 0003 is accepted. The configuration reference, operator guide, architecture page and design record describe both routes. Ticket: #50 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gh an egress proxy
… the Gateway never connects directly When the proxied leg fails, the Workload's CONNECT returns 502 with `Countersign-Request-ID` and a `Countersign-Error` that says why: `proxy_denied` for a 4xx answer from the egress proxy, `upstream_failure` for a 5xx answer, as a direct route reports a site it cannot reach, and `proxy_unavailable` when the proxy cannot be reached, resolves only inside `denied_networks`, runs out the deadline, closes unanswered or answers with something that is not HTTP. The proxy's own response body is never passed on, and the Gateway logs each failure with its request ID and error value. The runtime test's egress proxy learns to refuse with an error page of its own, to close unanswered, to answer with something that is not HTTP, to answer 503 when it cannot reach the site, and to tunnel a name it allows to a site whose certificate is for another name. In sidecar and shared mode the Workload reads each error value and no body. Two more Gateways per mode have no usable egress proxy, one with nothing listening at its address and one with its address inside `denied_networks`, while they can resolve and reach the site themselves. The site and the test proxy record every connection they accept, before any handshake, and every negative case asserts the site saw none, and for `denied_networks` that the proxy saw none either. Through the tunnel to the wrong certificate the request fails and the test grant authority, which now records each signature it is asked for, is asked for none. The troubleshooting table covers a 502 and its three values, and the operator guide's egress proxy section names them. Ticket: #51 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and the Gateway never connects directly
… in the Compose demo `compose_demo.py --squid` layers `compose.squid.yaml` over the default `compose.yaml`, which stays as it was. Squid runs from `ubuntu/squid:6.6-24.04_edge`, pinned by digest, as the demo's user with a read-only root and no capabilities. The Gateway shares the backend network with the Broker and Squid; the site moves to a network only Squid shares, in the network of a placeholder container, so stopping the site leaves its name and address behind. The Gateway's configuration names `http://squid:3128` as its egress proxy and its rule has no `private_networks`. `squid.conf` allows CONNECT only to site.test:9444 on the site's address, blocks metadata, link-local and loopback addresses, and does no TLS interception. With `--verify` the launcher checks that the Gateway logged the egress proxy and can neither resolve the site's name nor reach its address. From the approved request it correlates Squid's TCP_TUNNEL/200 line for the CONNECT to the site's address, the fingerprint the site returned and the approval, and the job page. Then, reusing that grant, it swaps Squid's rule to refuse the site and reconfigures it (502 proxy_denied, Squid logs TCP_DENIED/403 with no connection to the site), restores it and stops the site (502 upstream_failure, Squid logs TCP_TUNNEL/503), and stops Squid (502 proxy_unavailable). Each failure carries a request ID the Gateway logs with the same error value. The operator guide's egress proxy section gains example Squid ACLs taken from that configuration and the answers Squid 6.14 gives. The developer guide says how to run the variant and names the Squid version it was qualified against. Ticket: #52 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… Squid in the Compose demo
…iting for the deadline Review of spec #49, round 1: - The proxy's answer is checked as it arrives. Bytes that cannot be an HTTP answer fail at once with proxy_unavailable, and a non-2xx status line returns at once, even when the proxy keeps the connection open. Lines may end in a bare LF (RFC 9112, 2.2). - egress_proxy refuses port 0 at load. - The egress proxy startup line comes after the destinations pass their checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ery address, as private_networks does Review of spec #49, round 1: Squid's dst ACL matches when any one of a name's addresses is in its list, measured on the pinned Squid 6.14, so the example adds a deny on every address outside the prefix. The troubleshooting row says upstream_failure can also mean the rule's roots or revocation files did not load. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…addresses is another Review of spec #49, round 2: the operator guide's every-address pin was not in the configuration the Compose variant qualified. The demo's squid.conf now denies the site's name on every address but the site's, as the guide's example does for a private site. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
10 tasks done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #50
Closes #51
Closes #52
Summary
An operator whose network gives the Gateway no direct route can now name an egress proxy once, as
egress_proxyin the Gateway configuration, and every protected site is reached through it, while a rule with"egress": "direct"keeps its direct route. When the proxied leg fails, the Workload's 502 says whether the egress proxy refused, could not reach the site or could not be used, and the Gateway never connects directly instead.Where the route is chosen, in
connect_request(crates/countersign-gateway/src/runtime/mod.rs):runtime/config.rs):egress_proxyis anhttpURL with a host and a written port.https, credentials, port 0 andprivate_networkson a proxied rule each fail loading by name, inenvandservealike. ADR 0003 is nowaccepted.runtime/egress.rs): the Gateway resolves only the egress proxy, on every connection, and drops answers insidedenied_networks. It sendsCONNECT <site>:<port>. It reads the answer one byte at a time and gives up as soon as the bytes cannot be HTTP. A 4xx isproxy_denied, a 5xxupstream_failure, and anything elseproxy_unavailable.CONNECTproxy. Every negative case checks that the site saw no connection.compose_demo.py --squid, with a real Squid pinned by digest; the default walkthrough is unchanged.docs/develop.mdsays how to run it.dstmatches when any one of a name's addresses is in its list, and on the pinned Squid 6.14 the plain pin let a name through with one address outside it, then dialled that address. The demo'ssquid.confcarries the same deny.Extras for the owner
envaccepts configurations thatserverefuses at its destination checks. A wildcard rule withprivate_networks(refused as "private egress requires an exact host and prefix rule"), and anything elseConfig::policyrefuses, passesenvand stopsserve. This predates the change; every refusal this pull request adds is in both commands. An operator can send a Gateway's destinations through an egress proxy #50's criterion "envandserverefuse the same configurations" and the spec's story 15, "so that my launcher finds the mistake before the Gateway starts", read plainly, cover it.envwould run the destination checksserveruns)proxy_unavailablewhen the Workload's 120-second deadline passes. The code maps it through the same catch-all as the tested unreachable, closed and non-HTTP cases, but the deadline is fixed and CI runs the runtime test undertimeout 120. No source settles it.Evidence
egress_proxy, anhttpURL with a host and a portegress_configuration: anhttpegress proxy loadshttpsegress proxy URL and on one with credentials, and each error names the causeegress_configuration:https, credentials and a wrong shape are refused by nameegresswhose only value is"direct", and any other value fails loadingegress_configuration:egresstakes only"direct"private_networksand goes through the egress proxy, and the error names the rule and both fixesegress_configuration:private_networkson a proxied rule is refusedenvandserverefuse the same configurationsegress_configuration'srefused(), which runs every refusal through both commands and asserts the same one line from each (a pre-existing gap is under Extras for the owner)egress_proxyloads and behaves as before, and"egress": "direct"changes nothingruntime.py, unchanged and passing, the shared one with"egress": "direct"on its ruleegress_proxyset, the Workload reads the site through the test-only proxy, in sidecar and shared mode, and the site saw the approved certificateprivate_networksegress_proxy, a destination is still reached directlyruntime.pyandscripts/filter-sandbox-container.sh, on 5d76a8bacceptedand the glossary defines egress proxyegress_proxyandegressCONNECTrefusal gives 502proxy_denied, and the site saw no connectionfails_through_egress: a refusal isproxy_deniedupstream_failurefails_through_egress: a 5xx isupstream_failureproxy_unavailable, and the site saw nothing although the Gateway could reach itproxy_unavailablefails_through_egress: closed or not HTTP isproxy_unavailable; kept-open answers fail at oncedenied_networksgives 502proxy_unavailable, and neither proxy nor site saw a connectionCountersign-Request-IDand none carries the proxy's bodyfails_through_egress: a refusal isproxy_denied(every negative case asserts both)fails_through_egress: a refusal isproxy_denied(every negative case waits forGateway CONNECT <id> <value>)runtime.pyandscripts/filter-sandbox-container.sh, on 5d76a8bproxy_deniedandproxy_unavailable, andupstream_failureon a proxied routecompose_demo.py --squid --verify; the default--verifyanddocker compose configruns--verify: the Gateway has no direct route to the siteCONNECTonly to the site's hostname and port, and does no TLS interception--verify: Squid logged the tunnel to the site; the approved certificate reached the site--verifyrun, Squid logs a successfulCONNECT, the site's fingerprint equals the approved one, and the Workload read the responsecompose_demo.py --squid --verifyon 5d76a8bproxy_denied, and Squid logs the refusal--verify: the ACL swap givesproxy_denied; Squid logs the refusalupstream_failure--verify: a stopped site givesupstream_failureproxy_unavailable--verify: a stopped Squid givesproxy_unavailablesquid.confpins the site to every address--verifyrun of the variant is recorded on this ticketegress_configuration: anhttpegress proxy loadsAssertionError: http://proxy.test:3128egress_configuration:https, credentials and a wrong shape are refused by nameserveacceptedhttpsand kept running:TimeoutExpired ... timed out after 10 secondsegress_configuration: an empty port is refusedTimeoutExpired:servestarted on port 80 forhttp://proxy.test:egress_configuration: port 0 is refused (review)subprocess.TimeoutExpired: Command '[... 'serve', '.../egress.json']' timed out after 10 secondsforhttp://proxy.test:0egress_configuration:egresstakes only"direct"AssertionError(unknown fieldegress)egress_configuration:private_networkson a proxied rule is refusedservestarted:TimeoutExpiredegress_configuration: no startup line when the destination checks fail (review)AssertionError: ... Gateway egress proxy proxy.test:3128 carries the address policy of destinations: sitethenError: ... "private egress requires an exact host and prefix rule"('alice', b'curl: (22) The requested URL returned error: 502\n')sidecar/shared Gateway mTLS through the egress proxy passed, the proxy recorded exactlyCONNECT site.test:<port> HTTP/1.1, andgetent ahosts site.testexits 2 under the Gateway's environmentAssertionError: upstream leg survived cutoff('alice', b'curl: (22) ... 502\n')HTTPS_PROXY:AssertionError: The Gateway took the egress proxy from its environmentAssertionError: ['Gateway worker sandbox "namespaces": ...']fails_through_egress: a refusal isproxy_deniedcountersign-error: upstream_failurefails_through_egress: closed or not HTTP isproxy_unavailableupstream_failurefails_through_egress: a 5xx isupstream_failureproxy_unavailable, from the catch-allfails_through_egress: kept-open answers fail at once (review):banner,stub,lfTimeoutError: timed out(the Workload's 8-second read)proxy_unavailable,proxy_unavailable,proxy_denied('site.test:…', 'The site saw a connection', b'HTTP/1.1 200 OK…')'The site saw a connection', and with that check skipped,AssertionError: The egress proxy saw a connectionwrong.test:AssertionError: (0, b'd8bc83…'), and with that check skipped,A site with the wrong certificate got a signature--verify: Squid logged the tunnel to the siteRuntimeError: Timed out waiting for Squid's access log line for the Workload's CONNECT--verify: the Gateway has no direct route to the siteThe Gateway can resolve site.test without Squid; pointed atbroker:9443on purpose:The Gateway can connect to the site at 172.31.0.2 without Squid--verify: the startup line names Squidegress_proxyleft out:Timed out waiting for the Gateway's egress proxy startup line--verify: the ACL swap givesproxy_denied; Squid logs the refusalThe Workload's request succeeded where it should fail with proxy_denied; port 9445 expected on purpose:Squid did not log one refusal…--verify: the Gateway logs the same request ID…_wrongexpected on purpose:Timed out waiting for the Gateway's log line…--verify: a stopped site givesupstream_failure…should fail with upstream_failure;HIER_NONE/-expected on purpose for Squid's 503--verify: a stopped Squid givesproxy_unavailablecountersign-error: upstream_failuresquid.confpins the site to every address (review)squid.confbefore the change,site.testanswering 192.0.2.10 and 198.51.100.7:TCP_TUNNEL/503 0 CONNECT site.test:9444 - HIER_DIRECT/198.51.100.7TCP_DENIED/403 ... HIER_NONE/-in either order; 192.0.2.10 alone still tunnels (HIER_DIRECT/192.0.2.10)FATAL: Bungled /check/broken.conf line 4: acl tls_port prot 443. The plain pin,billinganswering one address inside and one outside: allowed, Squid dialledHIER_DIRECT/198.51.100.7squid -k parseexit 0;TCP_DENIED/403for the mixed answer in either order,HIER_DIRECT/10.30.0.5for the inside one; the block ACLs and deny rules equalsquid.conf's line for lineRuns on the integration branch's tip, 5d76a8b, on Linux arm64:
cargo fmt --all -- --check: clean.cargo clippy --workspace --all-targets --locked -- -D warnings: clean.cargo test --workspace --locked: 19 suites, 43 passed, 0 failed, 1 ignored.python3 crates/countersign-gateway/tests/runtime.py target/debug/countersign-gateway: exit 0 in 27 s (CI allows 120 s), printingsidecar Gateway mTLS passed,shared Gateway mTLS passed,sidecar Gateway mTLS through the egress proxy passed,sidecar Gateway without a usable egress proxy failed closed, the same two forshared, andfilter-killed grant worker fenced.scripts/filter-sandbox-container.sh: exit 0 in 33 s: the containment test, the same runtime-test lines under the filter sandbox, both named startup failures (refused to startfor each mode), and theclose_rangeunit (1 passed).mkdocs build --strict: clean.python3 -m unittest discover -s scripts -p 'test_git_hooks.py': 12 tests, OKdbus-run-session -- xvfb-run -a .venv/bin/python examples/demo-integrator/compose_demo.py --squid --verify --app <Companion>, with Squid 6.14 from the pinned image: exit 0. Its check lines are posted on A maintainer can qualify the Gateway against a real Squid in the Compose demo #52. The default walkthrough's--verifypassed on 17525c6. The defaultdocker compose configwas identical before and after (196 lines), andexamples/demo-integrator/compose.yamlis unchanged fromorigin/main.status: acceptedfor ADR 0003 and the glossary's**Egress proxy**. It also printed the configuration reference'segress_proxyandegressrows, and the operator guide's "Reach sites through an egress proxy" section with its "Who resolves the site's name" and "Who checks the site's addresses" rows and its ACL list. It found the troubleshooting row'sproxy_denied,proxy_unavailableand "On a route through the egress proxy, the proxy answered 5xx because it could not reach the site". It found the architecture page's "Egress proxy" trust-boundary row and the design record's "A destination takes one of two routes". It found the developer guide's "Qualify the Gateway against a real Squid" section, which names Squid 6.14.Not run:
pnpm lintandpnpm teston this tip, since no JavaScript changed after 17525c6, where they passed (the demo Integrator's tests: 8 passed). The x86-64 runtime test and the CI filter-sandbox jobs run on this push. Repeating the run on Fargate 1.4.0 is the #46 reporter's, out of the spec's scope.Merge Danger
Door: two-way. A revert removes
egress_proxyandegress; a Gateway configuration that sets either would then fail to load (unknown field), so revert only after taking them out.Blast Radius: the Gateway's route to a protected site, for operators who set
egress_proxy.Nothing deploys on the merge: the
countersign-gatewaycrate and theghcr.io/zeroae/countersign-gatewayimage ship at the next version tag. No flag: the change is an opt-in configuration field whose absence keeps today's behaviour. Live step at the merge, for whoever merges: comment on #46, pointing the reporter at the documented Compose configuration (docs/develop.md, "Qualify the Gateway against a real Squid", and the operator guide's "Reach sites through an egress proxy"), so they can repeat it on Fargate.Reviews
Correctness, Standards, Spec and Security, three rounds: 18 findings, each with its outcome; rounds 2 and 3 found 1 and 0
Correctness, round 1 (at
high, on 719bffc againstorigin/main)private_networks: Squid'sdstmatches any one address. Fixed: the example adds a deny on every address outside the prefix, measured on the pinned Squid.proxy_unavailable. Pushed back: the spec's failure table maps an egress proxy that "timed out" toproxy_unavailable.egress_proxyaccepts port 0. Fixed: refused at load.upstream_failurefrom a rule'srootsorrevocationfiles failing to load. Fixed: the row names them, for either route.config.policy().EgressProxy::parserepeats the URL-origin checks. Pushed back: the shared check answers "expected a URL origin", while the spec requiresegress_proxy's errors to name the cause; the overlap is five boolean checks.egresssits beside the moduleegress::. Pushed back: theegressleg predates this change and pairs withingress; the module is always called by path.Standards, round 1 (smells only; no written rule broken)
resolve_target's denied filter. Pushed back: three lines; the two lookups differ (the 64-answer cap,permitted, one pinned address against trying each), and ADR 0003 keeps the two routes' policies apart.Config::loadkeeps the error in its own words, beside theprivate_networkscheck that needs the parsed proxy.Egress::Defaultcannot be written. Pushed back: the enum names a rule's route; every unknown value,defaultincluded, fails loading, as tested.upstream_failure. Pushed back: one maps a status the Gateway chose, the other the egress proxy's answer (ADR 0003's table), each in one place.run. Pushed back:mainprintsrun's other errors in their existing form, which the startup-failure checks read; only the load error needed its own words, as An operator can send a Gateway's destinations through an egress proxy #50 asks.Spec, round 1
HTTP/1.fail at once withproxy_unavailable.Security, round 1 (the path list could not be read, so every changed path was judged as a control)
Spec, round 2 (fix round at
low, on 47cdf6a against 719bffc; Correctness, Standards and Security found nothing)squid.confdenies the site's name on every address but the site's, and the variant's--verifypassed again on 5d76a8b.Round 3 (fix round at
low, on 5d76a8b against 47cdf6a): Correctness, Standards, Spec and Security found nothing.Tracking
Epic: E2 Reach protected sites through an operator's egress proxy
Spec: #49 Reach protected sites through an operator's egress proxy
Ticket: #50 An operator can send a Gateway's destinations through an egress proxy
Ticket: #51 A Workload learns why a proxied connection failed, and the Gateway never connects directly
Ticket: #52 A maintainer can qualify the Gateway against a real Squid in the Compose demo
Flag: none
🤖 Generated with Claude Code