[Darkheir]

Description

Tip

This PR can be reviewed commit by commit
Only the last commit changes the behavior

storage_timeout_policy used to time the whole get_slice call from outside the storage, including S3's own retries and backoff sleeps, so throttling (SlowDown) turned into split failures (internal timeout on get_slice).

S3 storages now apply the policy to each attempt (request and body) inside their retry loop: waiting between attempts is no longer counted, and timed-out attempts are retried at most max_num_retries times. Other storages keep the existing wrapper.

Throttled requests also get 2 extra attempts (THROTTLING_EXTRA_ATTEMPTS), spaced by the usual exponential backoff, to outlast short throttling bursts.

Throttling (SlowDown) on a small slice, by how long the bucket keeps rejecting the request:

Throttling lasts	Before	Now
< 0.9s	✅ succeeds	✅ succeeds
0.9–1.75s	depends on jitter (attempt 4)	✅ succeeds
1.75–7.9s	❌ internal timeout on get_slice at 2s	✅ succeeds
7.9–15.75s	❌	depends on jitter (attempt 7)
> 15.75s	❌	❌ fails with the SlowDown error; the root's split retry still applies

Other cases:

Case	Before	Now
Other transient error (e.g. InternalError 500) lasting 1.9–3.75s	❌ cut at 2s	depends on jitter (5 attempts, the last at 1.88–3.75s)
Backoff sleep longer than the remaining budget	❌ cancelled while sleeping	✅ sleeps aren't counted
Large slice (20 MB, 22s budget) whose body breaks at 8s	Retry must finish within the remaining 14s	✅ retry gets a fresh 22s
An attempt hangs, max_num_retries: 0 (prod)	❌ fails at 2s	❌ fails at 2s (unchanged)
An attempt hangs, max_num_retries: 1	Whole call restarted from zero	✅ only the attempt is retried, with a fresh budget

How was this PR tested?

Added tests

[Copilot]

🟡 Changes recommended

Non-CLI initialization can lose configured S3 timeout protection, and SDK timeouts can be misreported as policy exhaustion.

2 open issues

• S3 resolver storages ignore the configured timeout policy · New
• SDK timeouts are misclassified as policy timeout exhaustion · New

What changed in this PR

This PR moves S3 slice timeouts inside the storage retry loop so backoff delays no longer consume the download deadline. Other storage backends retain the existing timeout wrapper.

Changes:

• Apply a fresh timeout to each S3 slice attempt, including response-body reads.
• Allow two extra attempts for throttling errors.
• Pass node-specific S3 options through CLI resolver setup and add retry tests.

File	Description
quickwit/​quickwit-storage/​src/​storage_resolver.rs	Accept node-specific S3 options.
quickwit/​quickwit-storage/​src/​object_storage/​s3_compatible_storage.rs	Implement per-attempt timeouts, metrics, and tests.
quickwit/​quickwit-storage/​src/​object_storage/​s3_compatible_storage_resolver.rs	Define and apply S3 node options.
quickwit/​quickwit-storage/​src/​object_storage/​mod.rs	Export S3 node options.
quickwit/​quickwit-storage/​src/​lib.rs	Expose S3 node options publicly.
quickwit/​quickwit-search/​src/​leaf.rs	Bypass the outer timeout wrapper for S3.
quickwit/​quickwit-config/​src/​node_config/​mod.rs	Extract single-attempt timeout calculation and document behavior.
quickwit/​quickwit-common/​src/​retry.rs	Add throttling classification, extra attempts, and tests.
quickwit/​quickwit-cli/​src/​tool.rs	Update resolver options for CLI tools.
quickwit/​quickwit-cli/​src/​service.rs	Pass S3 options during service startup.
quickwit/​quickwit-cli/​src/​lib.rs	Derive S3 options from node configuration.
quickwit/​quickwit-aws/​src/​retry.rs	Forward throttling classification through retry adapters.
quickwit/​quickwit-aws/​src/​error.rs	Identify SDK throttling errors from metadata.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-layer timeout ownership and shared AWS retry changes warrant runtime validation of cancellation and mixed-error exhaustion behavior before human approval.

No open issues 🎉

2 resolved since last review

• SDK timeouts are misclassified as policy timeout exhaustion
• S3 resolver storages ignore the configured timeout policy

_{Review effort: Balanced}