[alexluong]

Combines the attempt lookups of retries running at the same time into one log store query. Opt-in: RETRY_LOOKUP_BATCH_SIZE, off by default.

Stacked on #1154.

Why

Each retry looks up its previous attempt. On ClickHouse most of a bounded lookup's CPU (~8-10 of ~14 ms) is fixed per-query overhead, so a burst of retries spends most of its cost on overhead. One query for many retries pays it once: ~4 ms per retry at 10 per query, ~1 ms at 100.

Changes

1. Log store ListLatestAttempts: the latest attempt per (tenant, event, destination), not older than each key's bound.
   • ClickHouse: one query bounded by the oldest key's time, each key's bound applied to the result. Sets query_plan_max_limit_for_lazy_materialization so payloads are read only for returned rows; servers that reject the setting get the query without it.
   • Postgres: LATERAL over unnest, one index probe per key.
   • Memory: same semantics.
2. Retry lookup batcher: flushes at the batch size or RETRY_LOOKUP_BATCH_WAIT_MS after the first lookup. Splits a batch whose attempt times span more than 15 min, so one old retry doesn't widen the time range for the rest. A batch of one, and tasks without AttemptTime, use the single lookup.
3. Config: RETRY_LOOKUP_BATCH_SIZE (default 0 = off, max 100, capped at RETRY_MAX_CONCURRENCY), RETRY_LOOKUP_BATCH_WAIT_MS (default 50).

Notes

• Each retry still gets its own result and is acked, backed off or dead-lettered on its own.
• A failed batch query fails every retry in it; each backs off as usual.
• The lazy materialization setting needs ClickHouse 25.x; older servers run the query without it and read more.
• CI's -short run skips the ClickHouse and Postgres suites; they pass locally on ClickHouse 24.10 (without the setting), 25.8 (with it) and Postgres 16.

Results

~6,000 retries due within 20 s, ClickHouse 25.8 limited to 3 vCPU:

	batching off	batch size 50
lookup queries	8,416	1,286
ClickHouse CPU spent on lookups	~118 CPU-s	~31 CPU-s
ClickHouse CPU during the burst, median	2.78 cores	0.60 cores
ClickHouse CPU, p90	3.01 cores	0.96 cores
ClickHouse CPU, max	3.09 cores	2.52 cores
samples at the 3-core limit (≥ 2.7 cores)	20 %	0 %
events delivered	18,000 / 18,000	18,000 / 18,000

Setup

ClickHouse 25.8, 3 vCPU, 8 M rows in the attempts table (30 days, ~90 KB average payload). 3 Outpost instances (2 CPU, 2 GiB each), Redis-compatible retry queue, NATS internal queue. Retry schedule 1, 5, 300, 1800, ... s, LOG_BATCH_SIZE=50, LOG_BATCH_THRESHOLD_SECONDS=10, RETRY_VISIBILITY_TIMEOUT_SECONDS=30. 18,000 events (12 KB) in 60 s to one webhook returning 5xx for the first 20 s, then 200. Batching off = #1154. Lookup counts and per-query CPU from system.query_log, container CPU sampled every 5 s over the 320 s run; "during the burst" = samples above 0.2 cores. One run per build.

🤖 Generated with Claude Code