Skip to content

The runtime test's retirement cutoff passes on a busy arm64 host #32

Closed as duplicate

Description

@sodre

Parent

none

What to build

Find why the Gateway's runtime test fails on an arm64 workstation, and fix the cause: the product's cutoff, or the test's budget.

crates/countersign-gateway/tests/runtime.py:331, assert site.cutoff.wait(1), is the shared-mode check that a retirement notice cuts the upstream leg within one second. On 2026-11-19, on an aarch64 host (kernel 6.17, glibc 2.39, load average about 5), it failed in 7 of 8 runs of origin/main at ebbcba8 built apart, and in 5 of 6 runs on the branch of #28. Every failing run had printed sidecar Gateway mTLS passed first. CI's gateway-arm64 and curl jobs passed the same test on pull request #30, so the failure has so far been seen only on that host.

Two readings, not yet told apart: the cutoff on a retirement notice is slower than one second under load, which would be a product finding; or one second is too tight a budget for a busy host, which is the test's.

Acceptance criteria

  • The cause is named, with a measurement of how long the upstream leg takes to close after the retirement notice on a host where the test fails
  • If the cutoff itself is slow: it is fixed, with a test that was red before
  • If the budget is too tight: the test waits on a bound the design record's cutoff contract justifies, and says which
  • The runtime test passes 10 runs in a row on that arm64 host

Decided in: pull request #30 (ticket #28), Extras for the owner
Live step at merge: none

Activity

  1. added theissue type on Nov 20, 2026
  2. sodre commented on Nov 20, 2026

    @sodre
    MemberAuthor

    This was generated by AI during triage.

    Duplicate of #29: the same assertion, site.cutoff.wait(1) after Bob's retirement in the shared-mode part of the runtime test, failing on the same arm64 host. The cause was found during triage: the Gateway closes the upstream leg within about a millisecond, but the test site's /slow handler misses a leg that ends with recv raising BrokenPipeError. The agent brief on #29 carries the measurement and the acceptance criteria, including the ones from this issue that still apply: a red run with the upstream leg left open, and consecutive green runs on that host. Work continues there.

  3. added and removed
    triage:needs-triageMaintainer needs to evaluate this issue
    on Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions