[Darkheir]

Description

• Add bucket information for object_storage_requests_* metrics
  • Cardinality should stay low
• Keep the whole chain in the errors so we can identify the whole chain
• New metric for split retries on an other searcher
  • With a label identifying whether the retry was successful or not

How was this PR tested?

Describe how you tested this PR.

[Copilot]

🟡 Changes recommended

The cross-searcher retry metric also counts same-searcher retries, making its reported outcomes inaccurate for its stated purpose.

1 open issue

• Exclude same-searcher retries from rescue metrics · New

What changed in this PR

This PR improves object-store and search observability by identifying request targets, preserving error causes, and tracking split retry outcomes.

Changes:

• Add bucket or container labels to object-store request counters and duration metrics.
• Preserve error-chain messages in storage and search conversions, with regression tests.
• Add split retry counters labeled by outcome and tests for outcome accounting.

File	Description
quickwit/​quickwit-storage/​src/​opendal_storage/​base.rs	Pass operator names to request metrics.
quickwit/​quickwit-storage/​src/​object_storage/​s3_compatible_storage.rs	Label S3 request metrics with bucket names.
quickwit/​quickwit-storage/​src/​object_storage/​azure_blob_storage.rs	Label Azure request metrics with container names.
quickwit/​quickwit-storage/​src/​metrics.rs	Add bucket labels to request metric definitions.
quickwit/​quickwit-storage/​src/​metrics_wrappers.rs	Carry bucket labels through metric wrappers and helpers.
quickwit/​quickwit-storage/​src/​error.rs	Preserve storage error causes and test formatting.
quickwit/​quickwit-search/​src/​metrics.rs	Define split retry outcome counters.
quickwit/​quickwit-search/​src/​error.rs	Preserve search error causes and test formatting.
quickwit/​quickwit-search/​src/​cluster_client.rs	Record split retry outcomes and test accounting.

_{Review effort: Balanced}

[rdettai-sk]

do we need a metric with the number of retries at the s3 level (aws_retry)? I feel like that would be required if we want to tweak the retry sleep.

[Darkheir]

do we need a metric with the number of retries at the s3 level (aws_retry)? I feel like that would be required if we want to tweak the retry sleep.

I will add it