Skip to content

The runtime test's cutoff check after a retirement fails on one arm64 machine #29

Closed

Description

@sodre

Parent

none

What to build

crates/countersign-gateway/tests/runtime.py fails about two runs in three on one arm64 Linux machine (a DGX Spark, kernel 6.17), at the same line each time: assert site.cutoff.wait(1) after Bob's retirement in shared mode. The request has already failed, as it should; the test site has not seen its connection close within the second the assertion allows.

It fails the same way on main before #22, and it passes on GitHub's x86-64 and arm64 runners, so it is a timing fault of the test or of the supervisor's teardown, not of the worker sandbox. On that machine it hides real failures of the runtime test.

Find whether the upstream leg really closes late there or the one-second wait is too tight, and fix whichever it is.

Acceptance criteria

  • The cause is stated: a late close of the upstream leg after a retirement, or a wait too short for the host
  • The runtime test passes ten runs in a row on the machine where it failed
  • The cutoff assertions still fail when the upstream leg is left open (shown red once)

Found while building #22 Grant workers confine themselves under the namespace sandbox.

Activity

  1. added theissue type on Nov 20, 2026
  2. sodre commented on Nov 20, 2026

    @sodre
    MemberAuthor

    This was generated by AI during triage.

    Agent Brief

    Category: bug
    Summary: The Gateway runtime test's slow test site misses an upstream leg that ends in BrokenPipeError, so the cutoff check after Bob's retirement fails by chance

    Current behavior:
    In the shared-mode part of the Gateway runtime test, Bob's /slow request is in flight when his retirement notice is published. The test then expects the test site to see the upstream leg end within one second (site.cutoff.wait(1)). The site's /slow handler sets cutoff only on an empty read, ConnectionResetError or ssl.SSLEOFError. If its blocking recv on the TLS socket raises BrokenPipeError, the handler exits without setting cutoff, the exception goes to the server's handle_error (which ignores it quietly), and the assertion fails, however fast the Gateway was.

    Verified during triage on 2026-11-20 on the DGX Spark where this was reported (aarch64, kernel 6.17, load average about 6–8, debug build of 747ee18). I ran a copy of the test with its handler instrumented to log how each /slow leg ended:

    runs how Bob's upstream leg ended time from retirement notice to leg end curl exit
    passed 6 of 10 empty read about 1 ms 56 at about 2 ms
    failed 4 of 10 BrokenPipeError: [Errno 32] Broken pipe raised by recv about 1 ms 56 at about 2 ms

    On the failing runs, cutoff was still unset 11 seconds after the notice. The Gateway cuts both legs within about a millisecond on every run, so the cutoff on a retirement notice is not slow and the one-second budget is not too tight. The test site misses one of the ways the leg can end. This is the only cause seen. Every failure in #32 and #35, which are duplicates of this issue, fits it too.

    Desired behavior:
    The test site counts the upstream leg as ended however the transport ends it: an orderly close, a reset, a TLS EOF, a broken pipe, or any other OSError or ssl.SSLError from the read. The one remaining outcome, a read timeout with the leg still open, must leave cutoff unset, so the test still fails when the Gateway leaves the upstream leg open. The one-second wait stays as it is: the measurement above gives it a margin of about 1000×.

    Key interfaces:

    • The test site's /slow handler (the Site request handler's do_GET): its read loop decides whether cutoff is set. An upstream close must set it. A socket timeout (TimeoutError / socket.timeout) must not.
    • cutoff and slow_started on the test server: unchanged.

    Acceptance criteria:

    • A comment in the test or the pull request names the cause: recv raising BrokenPipeError on the test site's side of a leg the Gateway has already closed. Link this triage measurement or a fresh one.
    • The test site sets cutoff on every read outcome that means the leg ended, and only on those. A read that times out with the leg still open does not set it.
    • Shown red once: with the Gateway's upstream close on a retirement suppressed (or the test pointed at an upstream leg it keeps open), every cutoff assertion of the runtime test still fails. Say in the pull request how this was done.
    • The runtime test passes 10 consecutive runs on the arm64 host where it failed (the DGX Spark), under the namespace worker sandbox.
    • CI's runtime test jobs (x86-64 and arm64) pass.

    Out of scope:

    • Any change to the Gateway's cutoff code: the measurement shows it closes the leg within about a millisecond.
    • Widening the one-second budget, or any other timing budget in the runtime test.
    • Why OpenSSL reports a closed peer as EPIPE on a read on some runs and as EOF on others. Note it if it is quick to find; it is not needed for the fix.
  3. added and removed
    triage:needs-triageMaintainer needs to evaluate this issue
    on Nov 20, 2026
  4. self-assigned this
    on Nov 20, 2026
  5. added a commit that references this issue on Nov 20, 2026
    9cdb927
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions