Skip to content

feat: make retry timing explicit in docs and startup logs - #1157

Open
alexluong wants to merge 6 commits into
feat/retry-lookup-batchfrom
feat/retry-timing-warning
Open

alexluong wants to merge 6 commits into
feat/retry-lookup-batchfrom
feat/retry-timing-warning

Conversation

@alexluong

@alexluong alexluong commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Goal

Retry delays below LOG_BATCH_THRESHOLD_SECONDS are raised to it, so a schedule like 1,5,10 runs at 10, 10, 10 seconds. This makes that visible at startup and in the docs, so it doesn't read as a bug.

Changes

  1. Startup warning listing the configured retry delays that are raised, e.g. retry delays below LOG_BATCH_THRESHOLD_SECONDS (10s) are raised to it: 1s, 5s. A warning, not an error, so existing configs keep working.
  2. Event Delivery & Retries: retry delays are minimums and can run a few seconds late; the minimum is noted in both the Managed and Self-Hosted tabs.
  3. Configuration reference: LOG_BATCH_SIZE and LOG_BATCH_THRESHOLD_SECONDS rows (previously undocumented), including the retry minimum and 0 to remove it.

Notes

🤖 Generated with Claude Code


## Retry Timing

Retry delays are approximate. Outpost stores delivery attempts in batches, and a retry reads the attempt that triggered it, so a retry does not run before that attempt is stored. A delay shorter than `LOG_BATCH_THRESHOLD_SECONDS` (default `10`) is raised to it. Outpost logs a warning at startup listing the configured delays that are raised.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Too specific for the generic doc, needs to be in self-hosted section if needs to be more specific

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated!

Comment on lines +53 to +63
The effective delay of a retry is about:

- the larger of the configured delay and `LOG_BATCH_THRESHOLD_SECONDS`
- plus up to `RETRY_POLL_BACKOFF_MS` when you set it explicitly (the default, `0`, wakes the retry monitor when the next retry is due)
- plus up to `RETRY_LOOKUP_BATCH_WAIT_MS` when `RETRY_LOOKUP_BATCH_SIZE` is above `1`

If the attempt is still not stored when the retry runs, the retry waits about `LOG_BATCH_THRESHOLD_SECONDS` again (varied by up to 25%) and looks again.

For example, with `RETRY_SCHEDULE=1,5,10`, the defaults and `RETRY_LOOKUP_BATCH_SIZE=50`, each of the three retries runs about 10 seconds after its failed attempt, plus up to 50 ms for batching. With `RETRY_SCHEDULE=1,5,30` the third retry keeps its 30 seconds.

For tighter timing, set `RETRY_LOOKUP_BATCH_SIZE=0` and lower `LOG_BATCH_THRESHOLD_SECONDS`. A lower threshold flushes logs more often, which means more log store writes. Batching lookups helps high-volume deployments; at low volume, or on PostgreSQL, leave it disabled.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same, way to specific

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated!

RETRY_SCHEDULE=30,60,300,900,3600
```

Retry delays shorter than `LOG_BATCH_THRESHOLD_SECONDS` are raised to it, and Outpost logs a warning at startup listing them. See [Log Storage](/docs/outpost/self-hosting/configuration#log-storage) to shorten or remove this minimum.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These configs aren't explictly shows for managed so we can't refer to it here

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's in the self-hosted tab

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh!

@alexluong
alexluong marked this pull request as ready for review October 20, 2026 21:35

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants