Repository navigation
fix(ingress): retry evidence without reloading haproxy - #157
Merged
Merged
Conversation
One APPLY_PENDING flag covered evidence, the combined PEMs and the haproxy reload, so evidence that kept failing redid all three on every retry pass -- a SIGUSR2 every few minutes, each leaving old workers holding connections for up to TIMEOUT_CLIENT (24h by default). Track the reload and the evidence separately and retry only the step that failed. Signed-off-by: Kevin Wang <wy721@qq.com>
This was referenced Oct 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
One
APPLY_PENDINGflag covered evidence, the combined PEMs and the haproxy reload. Evidence that keeps failing (e.g. a guest agent error) redid all three on every retry pass: a SIGUSR2 every few minutes (backoff caps at 30 min), each leaving old workers holding connections for up toTIMEOUT_CLIENT(24h by default).What
Track
RELOAD_PENDINGandEVIDENCE_PENDINGseparately and retry only the step that failed. A pass with either pending still returns 3, so the retry backoff applies as before.Tests
Stubbed
run_passwith evidence failing twice: 4 passes → return codes 3, 3, 0, 0 and one reload (before: three reloads). Unit tests pass. Pebble end-to-end results will follow in a comment.Stacked on #156.