Skip to content

fix: requests never wait on an unavailable WebDecoy, and buffers stay bounded - #60

Merged
cport1 merged 1 commit into
mainfrom
fix/ingest-outage-resilience
Oct 2, 2026
Merged

cport1 merged 1 commit into
mainfrom
fix/ingest-outage-resilience

Conversation

@cport1

@cport1 cport1 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Problem: while WebDecoy was slow or unavailable:

  • protect() waited the full timeout (default 5s) on every suspicious request before failing open.
  • Every failure logged an error.
  • The violation buffer, AI referral counts and IP enrichment cache could grow without limit.
  • A 429 on AI referrals was treated as delivered.

Fix

  • One pause for every call: after a 429, a 5xx or no answer, the client pauses. It uses Retry-After when given, otherwise 1s doubling to 60s.
    • Calls during the pause throw WebDecoyUnavailableError without a request, so protect() returns ERROR at once (fail open, as before).
    • A 4xx answer is not an outage.
  • Logging: the failure that starts the pause logs at error; paused requests log at debug.
  • Bounded buffers:
    • Violations are held while paused (cap 1,000, oldest dropped) and sent on recovery.
    • The enrichment cache is capped at 10,000.
    • Referral counting stops adding new pairs, and stops triggering sends, while an unsent batch waits.
  • 429 on referrals: the batch is kept and retried under the same id.

Tests: 506 pass (9 new in src/outage.test.ts), plus tsc and eslint.

  • Mutation-checked: removing the pause gate, the quiet log, either cap, or the 429 handling each fails a test.

… bounded

- The client pauses all calls after a 429, a 5xx or no answer (Retry-After,
  else 1s doubling to 60s). While paused, calls throw
  WebDecoyUnavailableError without touching the network, so protect() fails
  open at once instead of waiting out the timeout on every request.
- The failure that starts a pause logs as an error; paused requests log at
  debug, so an outage does not flood the operator's logs.
- Violations are held while paused (capped at 1,000, oldest dropped) and sent
  when WebDecoy returns; the IP enrichment cache is capped at 10,000; AI
  referral counting stops adding pairs (and stops triggering sends) while an
  unsent batch waits.
- A 429 on AI referrals keeps the batch for retry under the same id.

Each protection is mutation-checked by src/outage.test.ts.
@cport1
cport1 merged commit f6a2f9f into main Oct 2, 2026
2 checks passed
@cport1
cport1 deleted the fix/ingest-outage-resilience branch October 2, 2026 17:32
@cport1 cport1 mentioned this pull request Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant