[alexluong]

Caps how many retries the retry scheduler runs at once and jitters its error backoff, so retries that come due together don't swamp the log store.

Monitor starts a goroutine per received task, with no limit. When thousands of retries come due within seconds, their lookups queue on the ClickHouse pool (10 connections, 10 s wait), most time out, back off exactly 30 s and come back together. The burst repeats at 30, 60, 120 and 240 s until the tasks are dead-lettered.

1. RETRY_MAX_CONCURRENCY (default 100 per process, 0 = unbounded). The monitor takes a slot before it receives, so tasks over the limit wait in the queue without using up receives.
2. Exec-error backoff is multiplied by a random factor in [0.75, 1.25), [0.75, 1) at the 15 min cap.
3. RETRY_MAX_RECEIVE_COUNT (default 5), for log store outages longer than the backoff window (~7.5 min at the 30 s default).
4. CLICKHOUSE_MAX_OPEN_CONNS (default: the client's 10), CLICKHOUSE_DIAL_TIMEOUT_SECONDS (default 10) and CLICKHOUSE_READ_TIMEOUT_SECONDS (default 30). clickhouse-go uses the dial timeout as the pool wait too.

Notes

• Unbounded → 100 by default. A retry execution is one lookup and one publish, so the log store pool binds first.
• Backoff is no longer exactly 30 s × 2^n.
• ClickHouse defaults unchanged.

🤖 Generated with Claude Code

[alexbouchardd]

Should we try to process in batch instead? Not so much a comment on this work specifically but wondering if the one-by-one is really the right approach

[alexbouchardd]

Ahh! #1155