Skip to content

fix: bound retry attempt lookup by the failed attempt time - #1152

Merged
alexluong merged 2 commits into
mainfrom
fix/retry-lookup-time-bound
Oct 21, 2026
Merged

alexluong merged 2 commits into
mainfrom
fix/retry-lookup-time-bound

Conversation

@alexluong

@alexluong alexluong commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Bounds the retry scheduler's attempt lookup by the failed attempt's time.

Problem

Before publishing a retry, the scheduler looks up the latest attempt for the event and destination:

SELECT ... FROM attempts
WHERE tenant_id IN (?) AND event_id IN (?) AND destination_id IN (?)
ORDER BY attempt_time DESC, attempt_id DESC LIMIT 2

ClickHouse sorts attempts by attempt_time, so without a time condition only the bloom skip indexes apply, and the event_id bloom keeps ~2.5% of all granules as false positives. Each lookup reads the key columns of those granules, so its cost grows with retention.

Fix

The latest attempt can't be older than the one that failed, so the retry task carries that attempt's time and the lookup starts there.

  1. RetryTask.AttemptTime, set when the retry is scheduled.
  2. The lookup sets TimeFilter.GTE = AttemptTime - 1s (storage precision). All log store drivers already support it.
  3. Tasks queued before the upgrade have no AttemptTime and run the unbounded query.

8M attempts over 30 days, one lookup:

read rows ClickHouse CPU
before 13.1 MiB 194k ~300 ms
after 0.04 MiB 552 ~14 ms

Notes

  • If the failed attempt isn't written to the log store yet, the lookup finds nothing and the task backs off until it is. Before, it found the attempt before that one and continued with a stale attempt number.

🤖 Generated with Claude Code

@alexluong
alexluong force-pushed the fix/scheduler-receipt-delete branch from cd7fce8 to 0c87e9b Compare October 21, 2026 15:31
Base automatically changed from fix/scheduler-receipt-delete to main October 21, 2026 15:31
@alexluong
alexluong force-pushed the fix/retry-lookup-time-bound branch from 8e822af to 7470a22 Compare October 21, 2026 15:32
@alexluong
alexluong merged commit ddd4b00 into main Oct 21, 2026
1 check passed
@alexluong
alexluong deleted the fix/retry-lookup-time-bound branch October 21, 2026 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants