Skip to content

fix: fail only the undecodable key in batched retry lookups - #1158

Open
alexluong wants to merge 4 commits into
feat/retry-lookup-batchfrom
fix/retry-lookup-row-errors
Open

alexluong wants to merge 4 commits into
feat/retry-lookup-batchfrom
fix/retry-lookup-row-errors

Conversation

@alexluong

@alexluong alexluong commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

A batched retry lookup fails as a whole when one returned row can't be decoded (event metadata that isn't a JSON object of strings, or response data that isn't JSON). Every retry in that query gets the error and is received again in a new batch, so one bad row counts a receive against every retry it shares a batch with and can push them into the retry DLQ. Without batching, only the retry whose own row is bad fails.

Outpost writes these columns itself, so a bad row needs an outside write or a bug. This change is defensive: a bad row now fails only its own retry. Query errors (timeout, store down) still fail the whole query, so they don't turn into extra per-key queries.

  1. driver.KeyErrors: ListLatestAttempts returns the other keys' results along with the per-key decode errors.
  2. ClickHouse and Postgres scans keep rows they can't decode, with the error, instead of aborting. ListAttempt on both still fails on such a row. Postgres reads event_metadata as raw JSON and decodes it after the scan, so a bad value no longer fails Scan itself.
  3. Batcher: a key error fails only that key's waiters.

🤖 Generated with Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants