Repository navigation
refactor!: move AccuracyModel and evidence to plan-selection; let the model accept outputs regardless of guarantee - #744
Merged
Conversation
zzylol
force-pushed
the
refactor/accuracy-model-plan-selection
branch
from
October 18, 2026 15:31
893964d to
af12b84
Compare
zzylol
force-pushed
the
refactor/accuracy-model-plan-selection
branch
2 times, most recently
from
October 18, 2026 16:10
fedc920 to
f8c3fbd
Compare
zzylol
force-pushed
the
refactor/accuracy-model-plan-selection
branch
from
October 18, 2026 16:48
8163234 to
2106009
Compare
Stage 3 is the only stage that uses the deployment's accuracy model (#509, #572 Q22). The AccuracyModel trait and DefaultAccuracyModel move from asap-logical-asap-optimization to asap_plan_selection::accuracy. Stage 1 calls the built-in analytical rules directly: the free functions accuracy::{local_guarantee, propagate, satisfies}. BREAKING CHANGE: asap_logical_asap_optimization::{AccuracyModel, DefaultAccuracyModel} are removed; use asap_plan_selection::{AccuracyModel, DefaultAccuracyModel}. AccuracyModel::local_guarantee and DefaultAccuracyModel::sketch_guarantee are removed (no caller).
…puts without a guarantee - Move AccuracyEvidenceProvider and NoAccuracyEvidence to asap_plan_selection::accuracy, next to AccuracyModel; OperandDomain and PropagationStats stay in Stage 1. - Rename DefaultAccuracyModel to BuiltinAccuracyModel. AccuracyModel::propagate gets a default body (the built-in rules), so a deployment model implements only satisfies. - Add AccuracyModel::accepts_output_without_guarantee (default false). Stage 3 asks it only when neither Stage 1 nor the evidence supplies a guarantee; an output it accepts is recorded with AcceptedAccuracy::accepted_without_guarantee, and one it does not is rejected with the new RejectionCause::NoAccuracyGuarantee (was AccuracyTargetMissed). - Document accuracy::estimators::local_guarantee and accuracy::composition::propagate, and drop their duplicate re-exports. - stage_pipeline writes accepted_without_guarantee; the dag-viewer accepts the new cause; the example fixture is regenerated. BREAKING CHANGE: AccuracyEvidenceProvider/NoAccuracyEvidence move to asap_plan_selection; DefaultAccuracyModel is now BuiltinAccuracyModel; a candidate whose output has no guarantee is rejected as no_accuracy_guarantee instead of accuracy_target_missed; AcceptedAccuracy gains a field.
…output regardless of its guarantee The hook now has priority over every guarantee: Stage 3 asks AccuracyModel::accepts_output_regardless_of_guarantee first for each query output, and an output it accepts skips the guarantee, Count-Min weight and evidence checks. Otherwise the check is unchanged (Stage 1 guarantee, then evidence, then AccuracyTargetMissed, or NoAccuracyGuarantee when neither gives one). The default returns false, so the built-in models behave as before. Tests pin that the hook accepts an output whose guarantee misses and one with no guarantee, in Stage 3 and end to end. Also drop the unused crate-root re-export of PropagationStats and re-indent the accuracy rejection reasons. BREAKING CHANGE: AccuracyModel::accepts_output_without_guarantee is now accepts_output_regardless_of_guarantee, and AcceptedAccuracy::accepted_without_guarantee (stage_pipeline JSON key accepted_without_guarantee) is now accepted_regardless_of_guarantee; asap_logical_asap_optimization::PropagationStats is now only at accuracy::PropagationStats.
…curacy model's hook The executor cannot run Count-Min (CmsWithHeap, Hydra Count-Min) over update weights not proven non-negative, and Stage 3's weight check is the only place that rejects such a plan. accepts_output_regardless_of_guarantee skipped it. The check now runs before the hook and rejects with the new failed-check cause RejectionCause::CountMinWeightsNotProvenNonNegative (count_min_weights_not_proven_non_negative); the hook still overrides every guarantee. A test also pins that the hook overrides guarantees that meet the target. BREAKING CHANGE: a plan rejected for unproven Count-Min weights now has cause count_min_weights_not_proven_non_negative instead of accuracy_target_missed.
zzylol
force-pushed
the
refactor/accuracy-model-plan-selection
branch
from
October 18, 2026 17:18
2106009 to
f442f0b
Compare
…Count-Min in the weights rejection Stage 3 composed through the model only for a ratio of relative estimates with enforced operand domains (domain_bound, ExactDivision); every other guarantee comes from Stage 1's built-in rules. The trait now has satisfies and accepts_output_regardless_of_guarantee, and domain_bound calls accuracy::composition::propagate directly. A Hydra (SharedMultiSubpopulation) summary is now named by its Hydra kind, so the unproven-weights reason reads "HydraCms needs non-negative update weights" rather than "Cms". Document which Count-Min forms the weight check covers and why: the executor refuses CmsWithHeap and HydraCms at plan build, and plain Count-Min fails at the first negative weight. BREAKING CHANGE: AccuracyModel::propagate is removed; a deployment model cannot change how guarantees compose.
zzylol
force-pushed
the
refactor/accuracy-model-plan-selection
branch
from
October 18, 2026 17:27
f442f0b to
c0bb156
Compare
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Issue #572 decided (Q22, following #509) that the deployment's accuracy model plugs into Stage 3 (plan selection) only. Stage 1 uses only the planner's built-in analytical rules. A candidate with no analytical rule is kept with
guarantee = None(not pruned), and Stage 3 decides on it.On main, two things do not match this:
AccuracyModeltrait,DefaultAccuracyModeland the deployment'sAccuracyEvidenceProviderstill live in the Stage 1 crate (asap-logical-asap-optimization), though only Stage 3 reads them.accuracy_target_missed, the same cause as a guarantee that misses.This PR finishes #572. Beyond Q22, it applies the user's decisions on the follow-ups:
BuiltinAccuracyModel;propagateis removed from the trait, because Stage 3 composed through it only for one quotient case.What
Move to Stage 3.
AccuracyModel,BuiltinAccuracyModel(wasDefaultAccuracyModel),AccuracyEvidenceProviderandNoAccuracyEvidencenow live inasap_plan_selection::accuracyand are re-exported at the crate root.OperandDomainandPropagationStatsstay in Stage 1, because its composition rules use them.Stage 1 has no model type. Its built-in rules are free functions, each with one path:
accuracy::estimators::local_guaranteeaccuracy::composition::propagateaccuracy::satisfiesStage 1's behavior is unchanged. It never read a deployment model, and it already kept a candidate with no rule with
guarantee = None. Tests on main pin this:combined_vector_quantiles_keep_unselected_summary_candidates(plannersummary_sharing.rs) anda_filter_or_group_key_over_an_estimate_has_no_guarantee(integration-testsper_computation_candidates_sql.rs).Simpler trait.
AccuracyModelnow hassatisfies(required) andaccepts_output_regardless_of_guarantee(defaultfalse).AccuracyModel::propagateis removed. Stage 3 called it in one place only:domain_bound, for a ratio of relative estimates with enforced operand domains (ExactDivision). That call now usesaccuracy::composition::propagatedirectly. Every other guarantee comes from Stage 1's built-in rules, so a deployment override had no effect elsewhere.AccuracyModel::local_guaranteeis removed: nothing called it through the trait.New hook, highest priority:
AccuracyModel::accepts_output_regardless_of_guarantee(output, target) -> bool, defaultfalse. For each query output, Stage 3's accuracy check now works in this order:RejectionCause::CountMinWeightsNotProvenNonNegative(count_min_weights_not_proven_non_negative). It wasAccuracyTargetMissedbefore.m, which is not declared a counter)". A Hydra structure is named by its Hydra kind, not "Cms".true, the output is accepted whatever its guarantee says (missing, met or missed), and no guarantee or evidence check runs for it.output_guarantee. If neither meets the target, the candidate is rejected:RejectionCause::NoAccuracyGuarantee(no_accuracy_guarantee) when neither source gives a guarantee;AccuracyTargetMissedwhen a guarantee misses.An output the hook accepts is recorded with
AcceptedAccuracy::accepted_regardless_of_guarantee = true. Its summary reads "metric rank; accepted by the deployment's accuracy model regardless of guarantee (Kll)".Behavior change (built-in models). The default hook accepts nothing, so selection is unchanged. Two causes change:
accuracy_target_missedand is nowno_accuracy_guarantee, with the reason "the output has no guarantee, and the deployment has no evidence for it, so it cannot meet target …";accuracy_target_missedand is nowcount_min_weights_not_proven_non_negative, with the same reason text.The updated tests show it:
count_min_over_signed_weights_is_rejected_as_invalid,count_min_over_samples_needs_a_declared_counterandcombined_estimates_need_evidence_about_the_output(plan-selection)combined_scalar_references_keep_summary_candidates(planner)planner-layering-example1.json: 38 candidates change cause (all wereaccuracy_target_missedbefore), and each of the 19 accepted candidates gainsaccepted_regardless_of_guarantee: falseon both queries. No candidate in that example reads Count-Min, so none moves tocount_min_weights_not_proven_non_negative. P1 is still selected.Devtools and viewer.
stage_pipelinewritesaccepted_regardless_of_guaranteeper accepted query. The dag-viewer'sFAILED_CHECK_CAUSESincludescount_min_weights_not_proven_non_negativeandno_accuracy_guarantee. The viewer's Python test and the devtools fixture check read the failed-check set from the viewer and fromRejectionCause::failed_a_check.Removed (no non-test caller):
AccuracyModel::local_guaranteeDefaultAccuracyModel::sketch_guaranteeaccuracy::{local_guarantee, propagate}(use the module paths)asap_logical_asap_optimization::PropagationStats(useaccuracy::PropagationStats)Closes #572
How
crates/plan-selection/src/accuracy.rsholds both traits and their defaults.BuiltinAccuracyModel::satisfiescalls Stage 1'saccuracy::satisfies. Stage 3's one composition (domain_bound) callsaccuracy::composition::propagatedirectly. Dependencies stay one-way: plan-selection depends on logical-asap-optimization, and Stage 1 depends on nothing in plan-selection.accuracy_check(plan-selectionlib.rs) returnsErr((RejectionCause, reason)). Per query it:CountMinWeightsNotProvenNonNegative;NoAccuracyGuaranteewhen neither exists, else asAccuracyTargetMissedwith today's reasons.impl DefaultAccuracyModel { fn additive … }block becomes plain private functions. Only the indentation changes (git diff -wshows only the removedimplandSelf::lines).local_guaranteeandpropagateget the doc comments the trait methods used to carry.Key code interfaces added / modified
crates/plan-selection/src/accuracy.rs(new module; re-exported fromcrates/plan-selection/src/lib.rs)crates/plan-selection/src/lib.rscrates/plan-selection/src/pass/mod.rscrates/logical-asap-optimization/src/accuracy/mod.rs,composition.rs,estimators/mod.rs,evidence.rs,lib.rscrates/devtools/src/bin/stage_pipeline.rs(document formatasap-stage-pipeline/v1)Usage (end to end)
From
crates/planner/tests/deployment_accuracy_model.rs. All three tests pass at the head SHA, and the ids, causes and summaries below come from running them.The first test in the same file plugs in an
ExactOnlyAccuracyModel(onlysatisfies, which accepts only exact guarantees; default hook) forquantile_over_time(0.99, lat[5m]). Stage 3 gets the same candidates P1–P3, with identical Stage 2 renderings. The KLL candidate P2, whose rank bound 0.00997 the built-in model accepts, is now rejected asAccuracyTargetMissed.Before this PR
AccuracyModel,DefaultAccuracyModelandAccuracyEvidenceProviderlived in Stage 1's crate, though only Stage 3 read them. Stage 1 called its own rules asDefaultAccuracyModel.propagate(...).propagate,satisfiesandlocal_guarantee, though Stage 3 calledpropagateonly for one quotient case. It could only grade guarantees. An output with no guarantee could be accepted only through evidence, and a guarantee that missed could not be overridden.accuracy_target_missed, which reads like a bound that missed.After this PR
asap_plan_selection::{AccuracyModel, BuiltinAccuracyModel, AccuracyEvidenceProvider, NoAccuracyEvidence}. Stage 1 exposes only its built-in rules, as free functions.satisfies, and may overrideaccepts_output_regardless_of_guarantee, which accepts an output directly, ahead of every guarantee check: missing, met or missed. It never overrides the executor's requirement that Count-Min weights be proven non-negative. Guarantees compose with the built-in rules only.no_accuracy_guarantee,count_min_weights_not_proven_non_negative).Migration for external code:
asap_logical_asap_optimization::{AccuracyModel, DefaultAccuracyModel}asap_plan_selection::{AccuracyModel, BuiltinAccuracyModel}asap_logical_asap_optimization::{AccuracyEvidenceProvider, NoAccuracyEvidence}(or::accuracy::…)asap_plan_selection::{AccuracyEvidenceProvider, NoAccuracyEvidence}asap_logical_asap_optimization::PropagationStatsasap_logical_asap_optimization::accuracy::PropagationStatsDefaultAccuracyModel.local_guarantee(f, q)asap_logical_asap_optimization::accuracy::estimators::local_guarantee(f, q)DefaultAccuracyModel.propagate(...)in Stage 1 codeasap_logical_asap_optimization::accuracy::composition::propagate(...)DefaultAccuracyModel.satisfies(g, t)asap_logical_asap_optimization::accuracy::satisfies(g, t)orBuiltinAccuracyModel.satisfies(g, t)impl AccuracyModel { fn local_guarantee …; fn propagate …; fn satisfies … }impl AccuracyModel { fn satisfies … }(accepts_output_regardless_of_guaranteeoptional;propagateandlocal_guaranteeremoved)models.accuracy.propagate(op, inputs, local, stats)asap_logical_asap_optimization::accuracy::composition::propagate(op, inputs, local, stats)accuracy_target_missedfor an output with no guaranteeno_accuracy_guaranteeaccuracy_target_missedfor unproven Count-Min weightscount_min_weights_not_proven_non_negativeAcceptedAccuracy { target_metric, accepted_metric, sketches }+ accepted_regardless_of_guarantee: boolLimitations and follow-ups:
accepted_metric: None), because no guarantee was checked.docs/develop_docs/asap-aware-mapping-architecture.md:69docs/develop_docs/asap-aware-mapping-contracts.md:64docs/develop_docs/end-to-end-accuracy-guarantees.md:56docs/develop_docs/extend-asap-aware-mapping.md:26docs/develop_docs/library-api.md:180anddocs/design_docs/architecture/input-output-workflow.md:21,235,242(DefaultAccuracyModelname)docs/design_docs/concepts/accuracy-models.md:143and:287(the Stage 1accuracymodule doc points readers there)docs/develop_docs/asap-aware-mapping-contracts.md:98(itsRejectionCausevariant list lackscount_min_weights_not_proven_non_negativeandno_accuracy_guarantee)docs/user_guide_docs/run-a-query.md:39-43: P2 is nowcount_min_weights_not_proven_non_negativeand P3 isno_accuracy_guarantee, notaccuracy_target_missedtools/dag-viewer/README.md(the cause list lackscount_min_weights_not_proven_non_negativeandno_accuracy_guarantee)Verification
All runs below are on the head, rebased on main b433a3e. After the rebase I regenerated the fixture with
stage_pipeline --example planner-layering-1. It is unchanged from the merged result, and the counts above are recounted against b433a3e's fixture.cargo nextest run -p asap-planner --test deployment_accuracy_model: 3 passed.…_accepts_outputs_without_a_guarantee: the hook accepts outputs with no guarantee; the default model rejects them.…_accepts_outputs_whose_guarantee_misses: the hook accepts P3, which the built-in model rejects asAccuracyTargetMissed. Every accepted output, including P1 and P2 whose guarantees meet the target, hasaccepted_regardless_of_guarantee == true. Moving the hook after the met-guarantee acceptance makes this test fail; I reverted the move.…_decides_only_in_plan_selection: the default hook leaves today's path unchanged.NoAccuracyGuaranteeand noaccepted_*field.unproven_count_min_weights_reject_even_an_accepting_model(written first; it failed to compile with no such cause): withmdeclared a Gauge, P2 (CmsWithHeap) is rejected asCountMinWeightsNotProvenNonNegative, with "not declared a counter", under an always-accepting hook and under the built-in model. Passes.unproven_hydra_count_min_weights_name_hydra_cms: P2 with its CmsWithHeap build replaced by a Hydra Count-Min, over a Gauge. The reason says "HydraCms needs non-negative update weights" and "not declared a counter". Written first; it failed with "Cms needs …" before the fix.the_accuracy_model_accepts_an_output_whatever_its_guarantee: the summary candidates of a quantile difference are rejected by the built-in model, asAccuracyTargetMissedwith evidence that misses and asNoAccuracyGuaranteewith no evidence. An accepting hook makes them valid in both cases. Passes.cargo nextest runover asap-plan-selection (whole crate), asap-planner (whole crate), devtoolsstage_pipeline, frontend-promqlunivmon_candidates, integration-testspromql_to_post_asap, and logical-asap-optimizationaccuracy::/per_computation_candidates::: 340 passed.count_min_over_samples_needs_a_declared_counter,combined_estimates_need_evidence_about_the_output), one planner test (combined_scalar_references_keep_summary_candidates) and the devtools fixture check (fixture regenerated withstage_pipeline --example planner-layering-1).a_bound_in_another_metric_misses_the_target, passed again once the evidence-miss reason kept its old wording.count_min_over_signed_weights_is_rejected_as_invalid(plan-selection).python3 -m unittest test_viewer(tools/dag-viewer): 37 passed.cargo fmt --all -- --check: clean.cargo clippy --all-targets --all-features --locked -- -D warningson logical-asap-optimization, plan-selection, frontend-promql, planner, integration-tests and devtools: clean.