You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
The runtime test's cutoff check after a retirement fails on one arm64 machine #29
crates/countersign-gateway/tests/runtime.py fails about two runs in three on one arm64 Linux machine (a DGX Spark, kernel 6.17), at the same line each time: assert site.cutoff.wait(1) after Bob's retirement in shared mode. The request has already failed, as it should; the test site has not seen its connection close within the second the assertion allows.
It fails the same way on main before #22, and it passes on GitHub's x86-64 and arm64 runners, so it is a timing fault of the test or of the supervisor's teardown, not of the worker sandbox. On that machine it hides real failures of the runtime test.
Find whether the upstream leg really closes late there or the one-second wait is too tight, and fix whichever it is.
Acceptance criteria
The cause is stated: a late close of the upstream leg after a retirement, or a wait too short for the host
The runtime test passes ten runs in a row on the machine where it failed
The cutoff assertions still fail when the upstream leg is left open (shown red once)
Found while building #22 Grant workers confine themselves under the namespace sandbox.
Category: bug Summary: The Gateway runtime test's slow test site misses an upstream leg that ends in BrokenPipeError, so the cutoff check after Bob's retirement fails by chance
Current behavior:
In the shared-mode part of the Gateway runtime test, Bob's /slow request is in flight when his retirement notice is published. The test then expects the test site to see the upstream leg end within one second (site.cutoff.wait(1)). The site's /slow handler sets cutoff only on an empty read, ConnectionResetError or ssl.SSLEOFError. If its blocking recv on the TLS socket raises BrokenPipeError, the handler exits without setting cutoff, the exception goes to the server's handle_error (which ignores it quietly), and the assertion fails, however fast the Gateway was.
Verified during triage on 2026-11-20 on the DGX Spark where this was reported (aarch64, kernel 6.17, load average about 6–8, debug build of 747ee18). I ran a copy of the test with its handler instrumented to log how each /slow leg ended:
runs
how Bob's upstream leg ended
time from retirement notice to leg end
curl exit
passed
6 of 10
empty read
about 1 ms
56 at about 2 ms
failed
4 of 10
BrokenPipeError: [Errno 32] Broken pipe raised by recv
about 1 ms
56 at about 2 ms
On the failing runs, cutoff was still unset 11 seconds after the notice. The Gateway cuts both legs within about a millisecond on every run, so the cutoff on a retirement notice is not slow and the one-second budget is not too tight. The test site misses one of the ways the leg can end. This is the only cause seen. Every failure in #32 and #35, which are duplicates of this issue, fits it too.
Desired behavior:
The test site counts the upstream leg as ended however the transport ends it: an orderly close, a reset, a TLS EOF, a broken pipe, or any other OSError or ssl.SSLError from the read. The one remaining outcome, a read timeout with the leg still open, must leave cutoff unset, so the test still fails when the Gateway leaves the upstream leg open. The one-second wait stays as it is: the measurement above gives it a margin of about 1000×.
Key interfaces:
The test site's /slow handler (the Site request handler's do_GET): its read loop decides whether cutoff is set. An upstream close must set it. A socket timeout (TimeoutError / socket.timeout) must not.
cutoff and slow_started on the test server: unchanged.
Acceptance criteria:
A comment in the test or the pull request names the cause: recv raising BrokenPipeError on the test site's side of a leg the Gateway has already closed. Link this triage measurement or a fresh one.
The test site sets cutoff on every read outcome that means the leg ended, and only on those. A read that times out with the leg still open does not set it.
Shown red once: with the Gateway's upstream close on a retirement suppressed (or the test pointed at an upstream leg it keeps open), every cutoff assertion of the runtime test still fails. Say in the pull request how this was done.
The runtime test passes 10 consecutive runs on the arm64 host where it failed (the DGX Spark), under the namespace worker sandbox.
CI's runtime test jobs (x86-64 and arm64) pass.
Out of scope:
Any change to the Gateway's cutoff code: the measurement shows it closes the leg within about a millisecond.
Widening the one-second budget, or any other timing budget in the runtime test.
Why OpenSSL reports a closed peer as EPIPE on a read on some runs and as EOF on others. Note it if it is quick to find; it is not needed for the fix.
Activity
feat(gateway): Every grant worker confines itself before it reads any…
sodre commentedon Nov 20, 2026
Agent Brief
Category: bug
Summary: The Gateway runtime test's slow test site misses an upstream leg that ends in
BrokenPipeError, so the cutoff check after Bob's retirement fails by chanceCurrent behavior:
In the shared-mode part of the Gateway runtime test, Bob's
/slowrequest is in flight when his retirement notice is published. The test then expects the test site to see the upstream leg end within one second (site.cutoff.wait(1)). The site's/slowhandler setscutoffonly on an empty read,ConnectionResetErrororssl.SSLEOFError. If its blockingrecvon the TLS socket raisesBrokenPipeError, the handler exits without settingcutoff, the exception goes to the server'shandle_error(which ignores it quietly), and the assertion fails, however fast the Gateway was.Verified during triage on 2026-11-20 on the DGX Spark where this was reported (aarch64, kernel 6.17, load average about 6–8, debug build of
747ee18). I ran a copy of the test with its handler instrumented to log how each/slowleg ended:BrokenPipeError: [Errno 32] Broken piperaised byrecvOn the failing runs,
cutoffwas still unset 11 seconds after the notice. The Gateway cuts both legs within about a millisecond on every run, so the cutoff on a retirement notice is not slow and the one-second budget is not too tight. The test site misses one of the ways the leg can end. This is the only cause seen. Every failure in #32 and #35, which are duplicates of this issue, fits it too.Desired behavior:
The test site counts the upstream leg as ended however the transport ends it: an orderly close, a reset, a TLS EOF, a broken pipe, or any other
OSErrororssl.SSLErrorfrom the read. The one remaining outcome, a read timeout with the leg still open, must leavecutoffunset, so the test still fails when the Gateway leaves the upstream leg open. The one-second wait stays as it is: the measurement above gives it a margin of about 1000×.Key interfaces:
/slowhandler (theSiterequest handler'sdo_GET): its read loop decides whethercutoffis set. An upstream close must set it. A socket timeout (TimeoutError/socket.timeout) must not.cutoffandslow_startedon the test server: unchanged.Acceptance criteria:
recvraisingBrokenPipeErroron the test site's side of a leg the Gateway has already closed. Link this triage measurement or a fresh one.cutoffon every read outcome that means the leg ended, and only on those. A read that times out with the leg still open does not set it.Out of scope:
EPIPEon a read on some runs and as EOF on others. Note it if it is quick to find; it is not needed for the fix.test(gateway): The runtime test sees every way a cut upstream leg can…