Stage 3 under-predicts CPU ~740×: measured costs price only sketch kernels, not scans, per-row work or merges #761

Problem, by example

ASAPQuery-backend ran the planner-layering PromQL examples on 2,000 series (90 minutes of data, 15 s samples, raw data retained). It ran Stage 3's selected plan with AWS Fargate prices and sketch-bench's measured operation costs, using a release build and perf sampling.

The sketch kernels that measured_operation_costs prices are only 3–7% of the measured CPU. perf breaks down the rest:

• Row handling in the executor: allocation and free 32%, row clones 11%, row drops 5%, grouping keys 9%, per-row memory accounting 9%, batch validation 2%.
• Raw-sample transfer: reading raw ranges back from Prometheus as JSON is about 15.5%, about 0.89 µs per sample.
• Scans and merges: Scan rows 5.6%, building kept panes 7.4%, merging and copying stored panes about 4%.

Because those costs are missing, Stage 3 underprices query-time plans that rebuild state from raw samples. With raw data retained it selected no ingestion-time outputs at all. The two quantile queries then rebuild 1,000 per-series KLLs at every evaluation, taking 57 and 313 ms in-process, against 31 and 62 ms when they read stored panes.

Cause

Stage 3's CPU estimate prices each operator by its kernel operations: sketch updates, merges and estimates, through measured or analytical per-operation costs. It does not price:

1. Scanning raw samples: per-sample decode and transfer from the data source, with the cost depending on where the raw data lives (local vs remote, JSON vs binary).
2. Per-row operator overhead: materializing, grouping, hashing and moving rows between operators, which is independent of the kernel.
3. Merging and copying stored state: reading N panes and merging them per evaluation, beyond the kernel's own merge cost.
4. Memory overheads: row layout, label strings and decoded buffers on top of the summary state.

Proposal

• Add per-sample and per-row cost terms to MeasuredOperationCost: scan per sample, per-row operator overhead, and merge or copy per stored pane. A runtime measures these and supplies them, so the Planner stays deployment-agnostic and only consumes numbers.
• Keep the analytical defaults for these terms, but make them realistic: derive them from a reference run instead of zero.
• Report predicted vs measured per plan in the stage_pipeline output, so the gap stays visible.

Acceptance

• With the runtime-measured terms supplied, Stage 3's predicted CPU for the example above is within 2× of the measured 53 core-seconds.
• A test pins that the scan and per-row terms change the choice between rebuilding from raw samples and reading stored panes, in the direction the measurements show.

Related

#754 (kept results almost never selected), #726 (sketch memory sized by full capacity), #734 (measured operation costs). ASAPQuery-backend design-doc PR #826 has the full measurement ("Overhead of the data plane" section).