Skip to content

The Gateway's runtime test passes or fails on the code, never by chance #35

Closed as duplicate

Description

@sodre

Parent

none

What to build

The Gateway's runtime test, crates/countersign-gateway/tests/runtime.py, passes or fails on the code under test and never by chance. Today one assertion fails intermittently: assert site.cutoff.wait(1), after broker.retired['bob'].set() in the shared-mode part, where Bob's retirement should cut his /slow request's upstream leg within a second. curl's request does end with an error, so the Gateway cut the downstream leg; the test site's handler does not set cutoff in time.

Measured on 2026-11-19 on an arm64 host (Linux 6.17, bubblewrap 0.9.0, Python 3.12) under moderate load: main at 02dc75f, its own test against its own binary, failed there 2 of 6 runs. The branch of #23 failed at a similar rate under both worker sandboxes, so the worker sandbox is not the cause. It passed on GitHub's runners in the one run of pull request #34.

First find which it is: the Gateway closing the upstream leg late after a retirement, which is a product defect in cutoff, or the test site missing the close (its handler sets cutoff only on an empty read, ConnectionResetError or ssl.SSLEOFError, and waits on a 25-second socket timeout otherwise).

Acceptance criteria

  • The cause of the intermittent failure is named, with the run that shows it
  • If the Gateway closes the upstream leg late after a retirement, that is fixed, with a test shown red once
  • If the test site misses the close, the site records every way the upstream leg can end
  • The runtime test passes 20 consecutive runs on a loaded arm64 host and on an x86-64 host

Found while building #23 (pull request #34).

Activity

  1. added theissue type on Nov 20, 2026
  2. sodre commented on Nov 20, 2026

    @sodre
    MemberAuthor

    This was generated by AI during triage.

    Duplicate of #29: the same assertion, site.cutoff.wait(1) after Bob's retirement in the shared-mode part of the runtime test, failing on the same arm64 host. The cause was found during triage: the Gateway closes the upstream leg within about a millisecond, but the test site's /slow handler misses a leg that ends with recv raising BrokenPipeError. The agent brief on #29 carries the measurement and the acceptance criteria, including the ones from this issue that still apply: a red run with the upstream leg left open, and consecutive green runs on that host. Work continues there.

  3. added and removed
    triage:needs-triageMaintainer needs to evaluate this issue
    on Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions