feat(bench): benefit experiment: Prometheus alone vs the Planner's plan, with measured results

[zzylol]

Why

The design doc's benefit experiment runs one workload twice on one deployment: (a) Prometheus alone answers every query exactly at query time; (b) the backend executes ASAPPlanner's selected plan, with Remote Write going to the data plane and everything the plan does not answer forwarded. The goal is absolute CPU, memory and latency for both runs, the error against the exact answer, and Stage 3's predicted usage next to the measured one. Before this PR the backend had a correctness end-to-end test (#822) but no way to run a workload at scale or measure anything.

What

• A new workspace binary, benefit (bench/), with three subcommands:
  • rules surveys a rule trace. It keeps the apiserver, etcd and kubelet rule groups and checks each rule alone: does the Planner plan it, does the data plane compile the plan, and does one evaluation over generated samples run? It writes rules.json, with the reason for every dropped rule.
  • plan writes the Planner workload, the selected plan, the same plan with nothing materialized (the correctness reference), and Stage 3's selection record.
  • experiment plans, runs the baseline and the ASAP run, compares them, and writes report.json and summary.md.
• A deterministic, seeded data generator (bench/src/generator.rs). It covers every series the rules read: Kubernetes control-plane families with realistic label sets, counters that only rise and reset with probability 1e-4 per sample, classic histograms (_bucket with le, _sum, _count) and gauges. It also covers the examples' http_requests_total and latency_ms. The scale is set by --scale (it multiplies the instance, node and peer dimensions), --sample-interval-ms, --duration-minutes and --seed.
• Deployment inputs come from the generator. DataWorkload facts per metric (series count, sample interval, declared type) and max_distinct_per_evaluation are computed from the generated series. Every series exists for the whole run, so the bound is the metric's series count.
• Price source flag --prices illustrative | aws-fargate-us-east-1, under cost model 1. The backend pin (58999df) predates the Fargate preset, so the flag reproduces ASAPPlanner main 4ef9f3e5's ResourceUnitCost::aws_fargate_us_east_1(speed) constants ($0.000011244/vCPU-s, $0.000001235/GiB-s). Replace it with the Planner function at the next re-pin.
• Measured operation costs: bench/inputs/measured_operation_costs.json, 138 entries. It is sketch-bench main a3753f2's study table (rqe-optimizer/results/autosketch-vs-asap-hydra/inputs/saturation/optimizer_cost/rqe_atomic_costs.json, last changed in 3fa808b, asap_sketchlib 0.3.0), converted by the pinned Planner's asap-devtools convert_rqe_atomic_costs. That tool skipped the exact accumulators and the Hydra rows. It is passed through --measured-operation-costs.
• control_plane::plan_and_select: plan_workload plus Stage 3's Selection, so the harness can report predicted usage.
• data_plane --execution-memory-limit-mib (default 64, the executor's own default). The hard-coded 64 MiB working-memory limit forwarded every Example 4b evaluation at 10× scale. The experiments use 4096.
• Results in bench/results/: raw JSON and a summary table per experiment, the rule survey, the joint-planning measurements, and a data-plane CPU profile.

How

One experiment does the following:

1. It plans the workload in-process: Stages 0–3 with generator facts and the chosen prices and measured costs.

2. It starts Prometheus 3.5.0, the release binary pinned by version and sha256, with --web.enable-remote-write-receiver. In the ASAP run it also starts the release data_plane with the plan. Each process is pinned with taskset.

3. It generates samples tick by tick (virtual time from 2026-01-01T00:00Z, as fast as the processes accept them). Each tick goes out as Remote Write in requests of at most 10,000 series: to Prometheus in the baseline, and to both Prometheus and the data plane in the ASAP run, because the data plane reads raw ranges from Prometheus.

4. After each tick it evaluates every query at each of its interval ticks in (previous tick, tick], from --eval-start-minutes on. These are instant queries at T = start + k·interval, as Prometheus rule evaluation would issue them. In the ASAP run they go to the data plane, which answers planned queries (x-asap-execution: plan) and forwards the rest.

5. A sampler reads /proc/<pid>/stat and VmRSS every 250 ms. CPU is the cumulative utime+stime at exit, startup included.

6. It compares each ASAP evaluation with the baseline's answer at the same T, and both with the Planner executor running the reference plan over the generated samples (compare::Reference). It checks every answer against the query's accuracy target:

   • exact targets: equal up to 1e-9 relative;
   • rank targets: the rank of the answer among the window's samples, inverse to Prometheus' interpolated quantile, within ε of the rank of Prometheus' answer.

   When the plan's labels differ from Prometheus', both answers are compared on the label names they share, and the row is counted under "Labels differ".

Key code interfaces added / modified

// control_plane/src/lib.rs
/// `plan_workload`, with Stage 3's selection record: every valid candidate's
/// predicted resource usage and cost, and why the others lost. Experiments
/// read it to compare predicted with measured usage.
pub fn plan_and_select(workload: &PlanningWorkload, inputs: &DeploymentInputs)
    -> Result<(InstalledPlan, Selection), PlanError>;          // added

// data_plane/src/precompute.rs
/// Sets the working memory every later execution may hold; an execution over
/// it fails (bad input to the deployment, not a bug) and its query is forwarded.
pub fn set_execution_memory_limit(bytes: usize);              // added
// data_plane binary: --execution-memory-limit-mib <MiB>      // added, default 64

# bench (binary `benefit`)
benefit rules      --trace rule_trace.jsonl [--accuracy <AccuracyTarget JSON>] [scale] [planning] --out-dir D
benefit plan       --workload examples|kube [--rules rules.json --only-rules a,b] [scale] [planning] --out-dir D
benefit experiment (same as plan) [--prometheus-bin --data-plane-bin --prometheus-cores --data-plane-cores
                   --eval-start-minutes --execution-memory-limit-mib --reuse-plan --reuse-runs] --out-dir D
  scale:    --scale N --sample-interval-ms --duration-minutes --seed --counter-reset-probability
  planning: --prices illustrative|aws-fargate-us-east-1 --measured-operation-costs F --no-raw-data

report.json: scale, series_per_metric, deployment_inputs, selection (selected id, node materializations, predicted usage of the selected and the nothing-materialized candidate, the five cheapest candidates), runs[] (per-process cpu_core_seconds, peak_rss_gib, mean_rss_gib, wall seconds, Prometheus version) and queries[] (evaluations, answered by plan, forward reasons, baseline = reference, plan = reference, meets target, labels differ, max abs/rel/rank error, p50/p99 latency per run).

Usage (the 50× examples run):

cargo build --release -p data_plane -p benefit
target/release/benefit experiment --workload examples --scale 50 \
  --prices aws-fargate-us-east-1 --measured-operation-costs bench/inputs/measured_operation_costs.json \
  --duration-minutes 90 --eval-start-minutes 10 \
  --prometheus-bin prometheus-3.5.0.linux-amd64/prometheus --prometheus-cores 0-3 --data-plane-cores 4-7 \
  --out-dir bench/results/examples-s50

Before this PR

There were no benefit numbers. Correctness was checked only in e2e_examples.rs, against a stand-in Prometheus, on a few series.

After this PR

These are measured results. All runs are on a 56-core x86 server, Prometheus on cores 0–3 and the data plane on cores 4–7, release builds. Prometheus is 3.5.0 (left-open ranges, matching the Planner). Prices are AWS Fargate us-east-1 under cost model 1 with the measured operation costs. The data is 90 minutes at one sample per 15 s per series, seed 1, and queries are evaluated from minute 10.

Correctness first. Every planned answer equals the Planner executor reference, and every baseline answer equals it too (exact targets, or within the rank target):

Experiment	Series	Evaluations	Answered by plan	Plan = reference	Baseline = reference	ASAP meets target
examples, 10×	2,000	1,052	1,052	1,052	1,052	1,052
examples, 50×	10,000	1,052	1,052	1,052	1,052	1,052
examples, 50×, no raw data (what-if)	10,000	1,052	1,047 (5 forwarded: panes not yet stored in the first hour)	1,047	1,052	1,052
kube rules, 10×	22,860	832	832	832	832	832
kube rules, 50×	114,300	832	832	832	832	832

Example 1's topk by (job) answers have the right values in all 481 evaluations of each run, but without the instance label Prometheus keeps (see Planner gaps). The quantile queries' worst rank error is 0.0100, at the 0.01 target. KLL returns the window's top sample, rank 1.0, while Prometheus interpolates at 0.99; the value difference against Prometheus' interpolated p99 is up to 1,587 ms.

Benefit, absolute numbers. CPU is in core-seconds and memory in GiB over the whole run:

Experiment	Run	Process	CPU core-s	Peak RSS GiB	Mean RSS GiB	Wall s
examples 10×	baseline	prometheus	21.69	0.093	0.087	19.3
	ASAP	prometheus	22.00	0.129	0.099	72.0
	ASAP	data_plane	53.35	0.336	0.111	72.0
examples 50×	baseline	prometheus	99.09	0.142	0.123	86.6
	ASAP	prometheus	103.54	0.304	0.166	361.8
	ASAP	data_plane	272.61	1.688	0.744	361.8
examples 50×, no raw data	baseline	prometheus	98.56	0.142	0.122	86.5
	ASAP	prometheus	32.99	0.135	0.120	277.9
	ASAP	data_plane	249.96	1.694	1.359	277.9
kube 10×	baseline	prometheus	144.59	0.235	0.200	119.9
	ASAP	prometheus	152.93	0.239	0.197	379.5
	ASAP	data_plane	265.87	1.280	0.860	379.5
kube 50×	baseline	prometheus	796.60	0.441	0.363	569.2
	ASAP	prometheus	883.99	0.656	0.436	2,218.6
	ASAP	data_plane	1,628.43	6.715	4.812	2,218.6

Latency p50 / p99 in ms, baseline vs ASAP (50× runs):

Query	Baseline	ASAP	ASAP, no raw data
example1 sum by (job) (rate(...[1m]))	46.7 / 72.3	248.7 / 328.1	168.8 / 1,700.6
example1 topk by (job) (10, sum_over_time(...[1m]))	46.8 / 75.1	238.0 / 304.5	134.6 / 202.7
example3b quantile_over_time(0.99, latency_ms[5m])	103.6 / 142.4	722.4 / 1,141.9	373.9 / 545.7
example4b quantile_over_time(0.99, latency_ms[60m])	406.9 / 531.1	4,699.1 / 6,399.1	436.9 / 666.0

Kubernetes rules, 50× (114,300 series), p50 / p99 ms:

Rule	Baseline	ASAP
cluster_verb_scope_le:apiserver_request_sli_duration_seconds_bucket:increase1h (every 180 s)	2,206.9 / 2,983.8	39,025.4 / 63,243.2
KubeAggregatedAPIErrors	6.7 / 11.3	255.6 / 561.0
KubeAggregatedAPIDown	9.9 / 16.3	665.1 / 1,054.1
KubeNodeNotReady	8.8 / 19.5	267.1 / 336.1
KubeletTooManyPods	19.6 / 55.1	159.1 / 240.9
KubeNodeEviction	36.3 / 65.3	880.1 / 1,250.9

The per-query tables for every experiment are in bench/results/*/summary.md.

Predicted vs measured (Stage 3, over the 90 minutes of data):

Experiment	Selected	Predicted CPU core-s	Measured data-plane CPU core-s	Predicted mean / peak memory GiB	Measured data-plane mean / peak RSS GiB	Predicted CPU, nothing materialized
examples 10×	P141-m8: 12 nodes MatAtQueryKeptAcrossEvaluations	0.072	53.35	0.000075 / 0.0269	0.111 / 0.336	0.128
examples 50×	P141-m8	0.361	272.61	0.00037 / 0.142	0.744 / 1.688	0.641
examples 50×, no raw data	P195-m40: 23 nodes MatAtIngestion	0.162	249.96	0.923 / 1.065	1.359 / 1.694	(no valid candidate without materialization)
kube 10×	P159-m485: 92 kept + 1 MatAtQuery	0.665	265.87	0.0026 / 0.0028	0.860 / 1.280	7.414
kube 50×	P159-m485	3.326	1,628.43	0.0128 / 0.0141	4.812 / 6.715	37.071

The prediction covers the plan's operator work only. It does not cover Remote Write decoding, the HTTP and JSON round trip of every raw range, or Prometheus' own ingestion. Measured data-plane CPU is 400–1,500× the prediction. Memory is closer only where state is materialized (the no-raw-data run: 0.92 GiB predicted mean vs 1.36 GiB measured).

Where the data plane's CPU goes (bench/results/profile-examples-s50/top30-self.txt, perf record -F 499 -g for 120 s of the 50× examples ASAP run, self time). About 40% is glibc malloc/free. About 15% is cloning and dropping Vec<asap_executor::values::Value> rows. summary::execute and execute_merge take about 7%, and serde_json raw-range decoding about 2%. The build has no frame pointers, so caller attribution is not reliable.

Result: no benefit at these scales. On the same deployment, the ASAP run uses more CPU, more memory and has higher latency than Prometheus alone in every configuration. Answers are correct throughout. Even when the data plane serves every query from ingestion-time panes (the no-raw-data what-if), Prometheus' query CPU drops by 65.6 core-s, but the data plane spends 250 core-s.

Workloads and rules

• Examples: the recurring queries of Examples 1, 3b and 4b, each at its own interval and accuracy target. Example 4's B3 has 3b's query. 3a (one-off reports over years) and 4a (monthly over years) need years of data and are left out.
• Kubernetes rules: the 75 distinct apiserver, etcd and kubelet rules of the recorded SRE-agent rule trace (groups whose name contains apiserver, etcd or kubelet). 13 are kept. The experiments run 6 of them, the largest set the Planner can plan jointly (see below): KubeAggregatedAPIErrors, KubeAggregatedAPIDown, KubeNodeNotReady, KubeletTooManyPods, KubeNodeEviction and cluster_verb_scope_le:apiserver_request_sli_duration_seconds_bucket:increase1h. All are exact, evaluated every 30 s or 180 s.

Rules dropped	Reason (from bench/results/kube-rules/rules.json)
17 (etcdNoLeader, etcdHighNumberOfFailedProposals, code_resource:…:rate5m ×2, code_verb:…:increase1h ×4, 7 POST/PUT burn-rate rules, KubeNodePressure, KubeNodeUnreachable)	Executor: regex label matchers (=~, !~) fail at run time ("unsupported scalar expression: Compare … op: Regex")
14 (KubeAPIErrorBudgetBurn ×4, KubeletPlegDurationHigh, the 30-day availability rules, …)	Read another recording rule's output; the generator writes raw series only
10 (every histogram_quantile rule: etcd fsync/commit/peer RTT, apiserver and kubelet quantile recordings, KubeletPodStartUpLatencyHigh)	Executor compile: "aggregate intent has no native implementation"
7 (the LIST/GET burn-rate rules)	Executor compile: "native Planner binding currently requires a unary operation or an explicit source"
7 (etcdMembersDown, etcdInsufficientMembers, etcdHighNumberOfFailedGRPCRequests ×2, etcdGRPCRequestsSlow, KubeClientCertificateExpiration ×2)	Executor compile: "grouping without requires resolved label columns" (sum without (…))
7 (KubeAPIDown, KubeletDown, KubeNodeReadinessFlapping, etcdHighNumberOfLeaderChanges, etcdDatabaseQuotaLowSpace, etcdExcessiveDatabaseGrowth, etcdDatabaseHighFragmentationRatio)	Executor compile: "per-entity aggregate requires an explicit entity binding" (absent, changes, last_over_time, predict_linear, subqueries)

Planner gaps found

1. Joint planning does not scale. The valid candidates of a workload are close to the product of its queries' (bench/results/kube-rules/joint-planning.json):

   • alone, the rule groups have A: 144, B: 1,792, C: 208 and D: 4 valid candidates;
   • B+D has 7,168 (19 s, 2.2 GB) and A+C 29,952 (41 s, 4.6 GB);
   • A+B and B+C run out of a 60 GB or 40 GB cap within two minutes;
   • all 13 kept rules were killed at 222 GB after 3 hours.

   So the full kept rule set cannot be planned.

2. Joint planning lost the kept-across-evaluations plans. With illustrative prices, each of the 7 POST burn-rate rules planned alone selects MatAtQueryKeptAcrossEvaluations; the seven selections cost 50.9 per second together, against 124.2 per second for the same plans with nothing materialized. Planned together, only 16 candidates are valid and P1 (nothing materialized) is selected at 122.6 per second. Those rules are now dropped by the regex gap, but the planning result stands.

3. topk by (job) drops the series labels. The PromQL frontend lowers the Example 1 query to output columns [job, topk_10]; Prometheus returns each top series with all its labels. The values match. with_promql_series_identity succeeds on this root, so the labels are lost at lowering, not in the backend.

4. The executor rejects regex matchers, without grouping, histogram_quantile and per-entity functions. It compiles or runs none of them (table above), and these make up most of the real rule set. A regex matcher passes compilation and fails only on the first evaluation, so rules probes one evaluation over generated samples.

5. Stage 3's CPU prediction is far below measurement, by 400–1,500× (table above). Its cost model counts operator work only, so a materialized candidate is predicted to save CPU that the deployment then spends on ingest and transport.

6. No candidate exists for the kube rules without raw data: every instant selector scans raw data. This is expected, but it means no configuration of these rules runs entirely from ingestion-time state.

Limitations

• Virtual time: 90 minutes of data is replayed as fast as the processes accept it, so CPU is total work, and the wall time of the ASAP run is longer than the baseline's.
• One machine runs both runs; the ASAP run's Prometheus also serves raw ranges, so its CPU includes that work.
• The prediction's period is the 90 minutes of data, while queries run only from minute 10.
• The profile was taken on the shared development machine (proportions only).
• The kube 50× comparison ran with --execution-memory-limit-mib 131072 and --reuse-runs. At 4096 MiB, the harness's nothing-materialized reference for the 1-hour histogram rule ran out of executor memory in 26 of 27 evaluations, while the plan itself answered all 27 exactly, equal to Prometheus.

What re-pinning to ASAPPlanner e12eca60 (#755) would simplify

• control_plane::plan_and_select, plan_selected_at_query_time and the -m id parsing in selection_summary become asap_planner::plan_installable, which returns InstallablePlan { dag, queries, stored_outputs, query_time_raw_windows, selection }.
• The data plane's SDS derivation (Sds::from_dag) and the query engine's walk from TimeRange to Scan with offsets (raw_input, program) would read stored_outputs { built_at, retention_ms, pane window } and query_time_raw_windows instead of re-deriving them.
• precompute::select and scan_below would use asap_executor::RawSampleSelector.
• The harness's --prices aws-fargate-us-east-1 reproduction becomes ResourceUnitCost::aws_fargate_us_east_1, from 4ef9f3e5, which is already older than e12eca60.

Verification

• cargo test --release -p benefit: the generator is deterministic for a seed, counters never decrease without a reset, and histogram buckets are cumulative with +Inf = _count; the rule parser separates metric names from recording-rule outputs.
• cargo fmt --all -- --check and cargo clippy --release -p benefit -p data_plane -p control_plane are clean. No CI ran (draft).
• End to end: the five experiments above, with every planned answer compared with the Planner executor reference and with Prometheus 3.5.0 (bench/results/*/report.json).