Repository navigation
Description
Problem Statement
An operator cannot run the Countersign Gateway in a network where the only way out is a proxy. In FINsight the Gateway is a sidecar in a Fargate task. The task reaches the Broker directly, but for an external HTTPS site it must go through Squid behind an internal load balancer. Squid authorizes the requested hostname and port and passes the TLS bytes through unread (#46).
The Gateway has no setting for that. Its supervisor resolves the site's name itself, checks every DNS answer against the site's address policy, and opens a direct TCP connection to the pinned address. With no direct route, the Workload's CONNECT returns 502 with Countersign-Error: upstream_failure, possibly only after the 120-second deadline.
The design record requires that order ("connect to the selected IP and port without a second DNS resolution"), so the Gateway cannot simply hand the name to a proxy without a decision about who then enforces the address policy.
Solution
The operator names an egress proxy in the Gateway configuration: the HTTP proxy the Gateway must go through to reach a protected site.
- One line,
egress_proxy, sends every destination through it. A rule that should connect directly says"egress": "direct". - On a proxied route the Gateway sends
CONNECTwith the site's name and port and resolves nothing. The egress proxy resolves the name, and its ACLs decide which names and addresses get a tunnel. - Everything else stays with Countersign: the site still needs its destination rule at the Gateway and the Broker, the user still approves, and the Gateway still verifies the site's certificate before the user's certificate is used.
- The Gateway never falls back to a direct connection. When the proxied leg fails, the Workload's response says whether the egress proxy refused, could not reach the site, or was itself unavailable.
ADR 0003 records the decision. A regression test runs on every pull request, and a Compose variant of the demo qualifies the Gateway against a real Squid.
User Stories
- As an operator whose only route out is a proxy, I want to set
egress_proxyonce, so that every protected site is reached through it. - As an operator, I want every destination to use the egress proxy by default, so that I do not repeat the route on each rule.
- As an operator with one internal site on a direct route, I want to set
"egress": "direct"on that rule, so that one Gateway serves both routes. - As an operator with no egress proxy, I want my configuration and my Gateway's behaviour unchanged, so that upgrading costs me nothing.
- As an operator, I want the Gateway to send the site's name to the egress proxy, so that my proxy can authorize by hostname.
- As an operator in a network without external DNS, I want the Gateway to resolve nothing on a proxied route, so that it works where only the proxy can resolve.
- As an operator, I want a private site to be reachable through the egress proxy with a rule that has no
private_networks, so that internal sites behind my proxy work too. - As an operator, I want a wildcard rule to work through the egress proxy, so that one rule covers a group of hosts my proxy allows.
- As an operator, I want the Gateway to refuse to start when a proxied rule has
private_networks, so that my config never claims an address check nobody performs. - As an operator, I want that error to name the rule and both fixes, so that I can correct it without reading the code.
- As an operator adding
egress_proxyto an existing config, I want every private rule that has not chosen a route to stop startup, so that I decide each one. - As an operator, I want an
httpsegress proxy URL refused at startup, so that I do not believe the proxy leg is encrypted when it is not. - As an operator, I want an egress proxy URL with credentials refused at startup, so that I do not believe the Gateway authenticates to the proxy when it does not.
- As an operator, I want an unknown
egressvalue to fail configuration loading, so that a typo cannot choose a route I did not mean. - As an operator, I want
countersign-gateway envto refuse the same configurations asserve, so that my launcher finds the mistake before the Gateway starts. - As an operator, I want startup to log which destinations leave their address policy to the egress proxy, so that I can see where each check happens.
- As an operator, I want the egress proxy's name resolved on every connection, so that a load balancer whose addresses change keeps working.
- As an operator, I want the egress proxy allowed on a private address, so that an internal load balancer works without extra configuration.
- As an operator, I want
denied_networksstill applied to the egress proxy's address, so that my block on a control-plane range holds for every socket the Gateway opens. - As an operator, I want a refusal by my proxy reported as
proxy_denied, so that I look at my proxy's ACLs. - As an operator, I want an unreachable or silent proxy reported as
proxy_unavailable, so that I look at the proxy and the network to it. - As an operator, I want the proxy's "could not reach the site" answer reported as
upstream_failure, so that a site outage reads the same on both routes. - As an operator, I want each of these to carry
Countersign-Request-ID, so that I can match a Workload's failure to the Gateway's log line. - As an operator, I want the operator guide to say how to set the egress proxy, so that I can configure it from the guide.
- As an operator, I want the guide to say where name resolution and address policy happen on each route, so that I know what my proxy is now responsible for.
- As an operator, I want example Squid ACLs in the guide, including blocks for metadata, link-local and loopback addresses, so that I do not leave open what the Gateway used to block.
- As an operator, I want the guide to show how to pin a private site to its prefixes at the proxy, so that I can keep the check
private_networksgave me. - As an operator, I want the configuration reference to describe
egress_proxyandegress, so that I can configure them from the reference alone. - As an operator, I want the troubleshooting table to cover the two new error values, so that I can act on them.
- As an operator running shared mode, I want the egress proxy to work there as in a sidecar, so that both deployment modes are available.
- As a Workload author, I want my client configuration unchanged, so that the same
HTTPS_PROXYand trust settings work on both routes. - As a Workload author, I want a failed proxied request to end with a status and an error value, never a hang until the deadline when the proxy has answered, so that my job fails fast.
- As a Workload author, I want the proxy's own error page kept from my job, so that I never parse a third party's HTML as a site response.
- As a user approving a grant, I want the prompt, the grant and the cutoff to work the same on a proxied route, so that my approval means the same thing.
- As a user who revokes a grant, I want a proxied connection cut off as a direct one is, so that revocation does not depend on the route.
- As a security reviewer, I want the Gateway never to fall back from the egress proxy to a direct connection, so that a proxy outage or refusal cannot become a way around it.
- As a security reviewer, I want the site's certificate verified against the hostname and the rule's roots before any signature, so that a tunnel to the wrong machine receives nothing.
- As a security reviewer, I want a site still to need its destination rule at the Gateway and the Broker, so that the egress proxy narrows what is reachable and never widens it.
- As a security reviewer, I want the egress proxy to carry only encrypted bytes, so that it never sees the user's traffic or the client handshake in the clear.
- As a security reviewer, I want the grant worker and the worker sandbox unchanged, so that the supervisor remains the only process that opens a socket.
- As a security reviewer, I want ADR 0003 to say which checks move to the egress proxy and which stay, so that the changed contract is checkable.
- As a security reviewer, I want the Gateway to ignore proxy environment variables on this leg, so that only the reviewed configuration chooses the route.
- As a reader of the design record, I want its resolution paragraph to name both routes, so that the record matches the product.
- As a Gateway maintainer, I want the runtime test to pass a request through a test-only
CONNECTproxy on every pull request, so that the proxied route cannot break unnoticed. - As a Gateway maintainer, I want that test to prove the site saw no connection when the proxy refused or was unreachable, so that "no direct fallback" is tested, not assumed.
- As a Gateway maintainer, I want that test to prove the proxy received the site's name and not an address, so that hostname authorization keeps working.
- As a Gateway maintainer, I want that test to run one Gateway with a proxied and a direct destination, so that the default and the opt-out are both exercised.
- As a Gateway maintainer, I want that test to reject a site certificate for the wrong hostname through an allowed tunnel, so that the certificate check is proven on the proxied route.
- As a Gateway maintainer, I want a Compose variant with a real Squid that I can run before a release, so that the Gateway is qualified against the proxy operators use.
- As a Gateway maintainer, I want that variant's check to correlate Squid's log, the certificate the site saw and the flag the Workload read from one run, so that a pass means the bytes took the proxied path.
- As a Gateway maintainer, I want the glossary to define egress proxy, so that code, docs and issues use one term.
- As the reporter of Validate Gateway sidecar mTLS through a mandatory Squid CONNECT proxy #46, I want a documented, passing Compose configuration, so that I can repeat it on Fargate 1.4.0 with the filter sandbox.
- As an Integrator, I want the demo walkthrough unchanged unless I ask for the Squid variant, so that trying Countersign stays as simple as it is.
Implementation Decisions
-
ADR 0003 binds this work. It is
proposed, and the first pull request lands it asacceptedtogether with the glossary entry. Use the glossary's term, egress proxy; avoid "upstream proxy", "outbound proxy" and "Squid" as the name of the concept. -
Configuration, top level. The Gateway configuration gains an optional
egress_proxy: a URL with thehttpscheme, a host and a port, such ashttp://proxy.internal:3128. Loading fails on anhttpsscheme and on a URL that carries credentials, and each error names the cause. -
Configuration, per destination. A destination rule gains an optional
egress, whose only value is"direct". Any other value fails loading, like every unknown field today.egress_proxyset: every destination goes through it, except a rule with"egress": "direct".egress_proxyabsent: every destination connects directly, as today."egress": "direct"is accepted and changes nothing.
-
private_networkson a proxied rule fails loading. The error names the rule and both fixes: remove the field, or set"egress": "direct". The existing refusal of a wildcard rule withprivate_networksis unchanged. -
Both commands validate.
envandserverefuse the same configurations. -
Both modes. Sidecar and shared mode take the same fields and behave alike.
-
The proxied leg. For a proxied destination the supervisor, where it opens the direct connection today:
- resolves the egress proxy's host through system DNS, on every connection, and refuses any answer inside
denied_networks.private_networksand the special-use blocks do not apply to the proxy's address; - connects and sends a plain-text
CONNECTwith the site's canonical name and port, with no proxy credentials; - reads a bounded response head. On a 2xx answer the socket takes the place the direct socket has today, and any bytes after the head belong to the tunnel.
The supervisor does not resolve the site's name on this route. The per-connection deadline covers the whole leg.
- resolves the egress proxy's host through system DNS, on every connection, and refuses any answer inside
-
What does not change. Destination policy, Broker admission, the grant, both leases, the grant worker, the worker sandbox, and the verification of the site's certificate against the hostname and the rule's roots and revocation evidence. Approval still comes before the connection. Each Workload
CONNECTopens its own tunnel; none is reused. -
No fallback. A proxied destination is never tried directly, whatever the failure.
-
Failures. Each returns 502 to the Workload with
Countersign-Request-ID, and the egress proxy's response body is never passed on.What happened Countersign-ErrorThe egress proxy refused the CONNECT(a 4xx answer)proxy_deniedThe egress proxy answered that it could not reach the site (a 5xx answer) upstream_failureThe egress proxy could not be reached, timed out, gave no usable answer, or resolved only to denied addresses proxy_unavailableThe Compose qualification confirms which answers a real Squid gives for an ACL refusal and for an unreachable site.
-
Environment. The Gateway takes the egress proxy only from its configuration. It reads no proxy environment variable for this leg, and the Broker leg stays direct.
-
Startup log. When an egress proxy is set, startup logs its host and port and the rule IDs of the destinations that leave their address policy to it.
-
The Compose variant. The demo gains an opt-in variant; the default walkthrough is unchanged.
- The Gateway shares a network with the Broker and Squid. The site is on a network only Squid shares.
- Squid runs from a pinned image, allows
CONNECTonly to the site's hostname and port, and does no TLS interception. - The demo's
--verifymode reads Squid's access log.
-
Documentation.
- The operator guide gains a section on reaching sites through an egress proxy. It says where name resolution and address policy happen on each route, and gives example Squid ACLs: allow by hostname and port, block metadata, link-local and loopback addresses, and pin a private site to its prefixes.
- The configuration reference describes
egress_proxyandegress. - The troubleshooting table covers
proxy_deniedandproxy_unavailable. - The architecture page's trust boundaries name the egress proxy.
- The design record's resolution paragraph names both routes.
- The developer guide says how to run the Squid variant and records the Squid version it was qualified against.
-
Flag: none. The change is an opt-in configuration field whose absence keeps today's behaviour, and the repository has no release-flag rule.
Testing Decisions
-
What makes a good test here: it checks what a Workload, an operator, the egress proxy and the site can each observe. Assert the status and headers the Workload gets, the request line the proxy received, the connections and certificate the site saw, and what the Gateway logs and how it refuses a configuration. Never assert how the supervisor builds the request or parses the answer.
-
Seam 1: the real Gateway binary, end to end. Extend the existing runtime test, which runs the real
countersign-gateway serveagainst a test grant authority and an mTLS site. It gains a small test-onlyCONNECTproxy that records each request line and can be told to allow, refuse or be absent. No new seam is added below it. Cases:- Allowed. The Workload reads the site's response through the proxied route. The site saw the approved client certificate. The proxy's recorded request names the site's hostname and port, and the Gateway had no way to resolve that name itself.
- Refused. The Workload gets 502 with
proxy_denied. The site saw no connection. - Site unreachable from the proxy. The Workload gets 502 with
upstream_failure. - Proxy unreachable. The Workload gets 502 with
proxy_unavailable. The site saw no connection, although the Gateway could have reached it directly. - Wrong hostname. The proxy tunnels to a site whose certificate is for another name. The request fails and no client-certificate signature is made.
- Default and opt-out together. One Gateway serves a proxied destination and a direct one with
private_networks; each takes its own route. - Cutoff. A retired grant cuts a proxied connection as it cuts a direct one.
- Configuration loading.
private_networkson a proxied rule, anhttpsproxy URL, a proxy URL with credentials and an unknownegressvalue each fail, and the first three name their cause. A configuration withoutegress_proxyloads as before. - Startup log. It names the egress proxy and the destinations that leave address policy to it.
It runs in both modes and under both worker sandboxes wherever the runtime test already does.
-
Seam 2: the Compose demo with a real Squid. The demo's
--verifymode drives the real Companion and browser against the variant, by hand before a release. No workflow runs it, as none runs the Compose demo today. From one run it correlates three observations: Squid's log line for theCONNECTto the site's hostname and port, the certificate fingerprint the site saw, and the flag the Workload read. It then checks a destination Squid refuses, and a stopped Squid; in both the Workload fails and the site sees no connection. -
Prior art: the runtime test's site and grant-authority fixtures, its configuration-loading check and its startup-failure check, for seam 1. The Compose demo's
--verifymode for seam 2.
Out of Scope
- Repeating the run on Fargate 1.4.0 with the filter sandbox. The reporter does that from the documented Compose configuration; Countersign's part ends at Compose for now.
- Proxy authentication and a TLS-wrapped proxy leg.
- More than one egress proxy, or a proxy URL per rule.
- Letting the egress proxy alone decide which sites a Gateway admits, with no destination list (An operator can let the egress proxy decide which sites a Gateway admits #48).
- Sending the Broker leg through the egress proxy.
- Running the Squid qualification in a hosted workflow.
- Any change to the grant worker, the worker sandbox, the sidecar CA or the approval prompt.
Further Notes
- Origin: Validate Gateway sidecar mTLS through a mandatory Squid CONNECT proxy #46, a site report from the FINsight deployment, triaged ready-for-human until this design settled who enforces the address policy on a proxied route. Validate Gateway sidecar mTLS through a mandatory Squid CONNECT proxy #46 is done when the Compose configuration passes and is documented.
- What the operator takes on, and the documentation must say plainly:
- On a proxied route the proxy's ACLs are the address policy. The Gateway no longer blocks metadata, link-local or loopback addresses for those sites.
- The Gateway cannot stop the egress proxy from opening a connection to any address its ACLs allow. Such a tunnel receives no signature unless the machine presents a valid certificate for the hostname.
- The Gateway refuses a wildcard rule for a private site on a direct route. Through the egress proxy it cannot tell that a site is private, so a wildcard rule reaches private hosts if the proxy allows them.
- A user can approve and the request then fail at the egress proxy, as it can today at an unreachable site.
- Options considered and rejected, recorded in ADR 0003:
- The Gateway resolves and sends
CONNECTwith the pinned address: a proxy that authorizes by hostname cannot, and the Gateway needs DNS the network may not give it. - The Gateway resolves and checks, then sends the name: the proxy resolves again, so the check does not bind the address it dials.
private_networksimplying a direct route, or being ignored on a proxied one: the first fails only after a user has approved, the second leaves a field that promises a check nobody performs.
- The Gateway resolves and sends
- Risk the tests must catch: a failure path that retries directly. Every negative case in seam 1 asserts what the site saw, not only what the Workload got.
Activity
docs(docs): A proposed ADR says who enforces a site's address policy …