Repository navigation
Keep the reflector's keep-alive connections open between the broker's requests - #168
Open
shawnburke wants to merge 1 commit into
Open
shawnburke wants to merge 1 commit into
shawnburke wants to merge 1 commit into
Conversation
… requests The HTTP server set ReadTimeout without IdleTimeout, and net/http falls back to ReadTimeout as the keep-alive idle limit. One second is shorter than the gaps between consecutive requests from the broker, which keeps its connections open by default (Node 20's agent), so it regularly wrote the next request into a socket this server had just closed. Each one surfaced as "socket hang up" ECONNRESET and a 500ms retry; a 30-minute window from one relay showed 4,774 such retries, and the 7 that exhausted their retries included 3 of the dispatcher's own /__axon/info liveness probes. Set IdleTimeout explicitly to five minutes so an idle connection outlives any realistic gap, and cover it with a test that reuses a connection after sitting idle past ReadTimeout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
shawnburke
marked this pull request as ready for review
October 15, 2026 00:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The agent's HTTP server (the reflector the broker talks to on 127.0.0.1) set
ReadTimeout: 1sand noIdleTimeout.net/httpusesReadTimeoutas the keep-alive idle limit whenIdleTimeoutis zero, so the server closed any keep-alive connection that sat idle for one second.The broker keeps its local connections open (Node 20's default agent), and the gap between its requests is often longer than a second. It regularly wrote the next request into a socket the server had just closed and got
ECONNRESET/ "socket hang up", then retried after 500ms.From one Fiserv relay (agent 0.3.4, 18:00 to 18:30 UTC on 2026-10-14):
/__axon/infoliveness probesA failed liveness probe is what the dispatcher reports as "agent unresponsive", so this churn feeds false negatives on the Cortex side as well as latency on relayed calls.
What
IdleTimeoutexplicitly to five minutes on the server.ReadTimeoutis unchanged.TestServer_KeepsKeepAliveConnectionIdlePastReadTimeout: send a request on a raw connection, wait longer thanReadTimeout, send another on the same connection, and expect a response. It fails onmainwithunexpected EOF.Verified
go vet ./server/http/clean;go test ./...inagent/green (14 packages).Context
Found while investigating CD-662 (Fiserv GitLab relay timeouts). This is a side issue there, not the cause of the outage, but it is independent and worth fixing on its own.
🤖 Generated with Claude Code