Repository navigation
Conversation
The ingestion side of a stored definition, Scan -> TimeShift* -> TimeRange(pane) -> SummaryAgg, runs per sample instead of over each pane's buffered rows: - per series, once: the executor's RawSampleSelector turns the series' first sample into its input row (applying the Scan's matchers), and the row's group columns give its group; cached by label set; - per sample: the open pane's state for that group is updated in place with the updater the executor's batch build uses for the family (create_planner_accumulator), with the same value and timestamp; - per group, once: the definition's compiled build runs over the group's first row, and its output row is the template each closed pane's row is made from (pane timestamp and state replaced). A build of another shape (row filter, keyed or shared-grid family, operators between the range and the summary, matchers reading the value or time) keeps the batch build. A parity test checks every family the examples store (counter rate, sum, KLL at two pane widths) against the executor's batch build of the same samples: exact states byte for byte, KLL by quantile (KLL seeds its compaction randomly per sketch).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 3/4. Base: #829.
Why
Pane builds at ingestion buffered every sample, then turned each into a row of
Values with its labels and JSON series identity, grouped rows by byte-vector keys and built states. That cost 10.5 µs per sample, against ASAPQuery's 2.3 µs, which updates each pane's accumulator in place per sample. User decision: ingestion updates summary state directly per sample, as the DAG's ingestion side compiled per sample (design doc, "Ingestion: the DAG run per sample"). There is no hand-written query path.What
A stored definition whose build is
Scan → TimeShift* → TimeRange(pane) → SummaryAggruns per sample:RawSampleSelectorturns the series' first sample into its input row, which applies theScan's label matchers. The row's group columns give its group: for a per-seriesSummaryAgg, every column but the value and the time; forby (...), its keys. The result is cached by label set.SummaryAgguses (asap_executor::factory::create_planner_accumulator), with the same value and timestamp, in arrival order. Counters keep the batch build's rule that times strictly increase within a group.A build of another shape keeps the batch build from buffered samples:
TimeRangeand theSummaryAgg;Scanmatchers that read the value or time.Each worker logs which definitions run per sample. The sharded part merge of the previous PR is unchanged.
Before this PR
Each pane's samples were buffered per worker and built at close with the executor's batch build, which made one row per sample.
After this PR
No per-pane batch is built from buffered samples for these shapes: in the examples' no-raw plans, all 23 ingestion-time outputs (counter rate and sum over 10 s panes, KLL over 60 s and 600 s panes). Only the group template is built, once per group.
Verification
cargo test --workspacepasses, includinge2e_examples(9),range,kept.per_sample::per_sample_panes_equal_the_batch_build. For Example 1 (rate, sum), Example 3b (KLL, 1-minute panes) and Example 4b (KLL, 10-minute panes), on 1 and 4 shards, every pane built per sample equals the executor's batch build of the same samples, row by row. Exact accumulator states are compared by their serialized bytes. KLL states are compared by their quantiles at every percent (exact below k). KLL seeds its compaction randomly per sketch, so two batch builds of the same samples are not byte-identical either. A mutation that drops every other sample makes both tests fail.per_sample::per_sample_panes_apply_label_matchers.Benefit harness
Same setup as #828.
s10
s10-noraw
s50
s50-noraw
With raw data retained the selected plan builds nothing at ingestion time (12 outputs kept across evaluations), so those runs are unchanged within noise. Without raw data, all 23 ingestion-time outputs run per sample. The remaining data-plane CPU is mostly query-time evaluation in the executor's row engine (design doc, "Execution engine"), which this PR does not change. "Prometheus alone" is the baseline of the pr2 run.
Planner gaps found
create_planner_accumulator) but not the lowering of aSummaryAggnode to its input columns (value, time, groups), nor its output assembly (scope_timestampispub(crate)). The backend reads the groups from the plan'sSummaryAggpayload (the same rule as the executor's lowering), and gets the layout by running the compiled build once per group. The parity test guards the copy of the rule. Smallest API proposal, inasap-executor: