Skip to content

refactor!: move AccuracyModel and evidence to plan-selection; let the model accept outputs regardless of guarantee - #744

Merged
zzylol merged 6 commits into
mainfrom
refactor/accuracy-model-plan-selection
Oct 18, 2026
Merged

zzylol merged 6 commits into
mainfrom
refactor/accuracy-model-plan-selection

Conversation

@zzylol

@zzylol zzylol commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Why

Issue #572 decided (Q22, following #509) that the deployment's accuracy model plugs into Stage 3 (plan selection) only. Stage 1 uses only the planner's built-in analytical rules. A candidate with no analytical rule is kept with guarantee = None (not pruned), and Stage 3 decides on it.

On main, two things do not match this:

  • The pluggable AccuracyModel trait, DefaultAccuracyModel and the deployment's AccuracyEvidenceProvider still live in the Stage 1 crate (asap-logical-asap-optimization), though only Stage 3 reads them.
  • A deployment's model can only grade guarantees. It cannot accept an output directly from its own measurements. An output with no guarantee can only be accepted through evidence; otherwise it is rejected as accuracy_target_missed, the same cause as a guarantee that misses.

This PR finishes #572. Beyond Q22, it applies the user's decisions on the follow-ups:

  • the evidence provider moves with the model;
  • the model gets a hook that accepts an output regardless of its guarantee, ahead of every guarantee check;
  • the default model is renamed BuiltinAccuracyModel;
  • propagate is removed from the trait, because Stage 3 composed through it only for one quotient case.

What

  • Move to Stage 3. AccuracyModel, BuiltinAccuracyModel (was DefaultAccuracyModel), AccuracyEvidenceProvider and NoAccuracyEvidence now live in asap_plan_selection::accuracy and are re-exported at the crate root. OperandDomain and PropagationStats stay in Stage 1, because its composition rules use them.

  • Stage 1 has no model type. Its built-in rules are free functions, each with one path:

    • accuracy::estimators::local_guarantee
    • accuracy::composition::propagate
    • accuracy::satisfies

    Stage 1's behavior is unchanged. It never read a deployment model, and it already kept a candidate with no rule with guarantee = None. Tests on main pin this: combined_vector_quantiles_keep_unselected_summary_candidates (planner summary_sharing.rs) and a_filter_or_group_key_over_an_estimate_has_no_guarantee (integration-tests per_computation_candidates_sql.rs).

  • Simpler trait. AccuracyModel now has satisfies (required) and accepts_output_regardless_of_guarantee (default false).

    • AccuracyModel::propagate is removed. Stage 3 called it in one place only: domain_bound, for a ratio of relative estimates with enforced operand domains (ExactDivision). That call now uses accuracy::composition::propagate directly. Every other guarantee comes from Stage 1's built-in rules, so a deployment override had no effect elsewhere.
    • AccuracyModel::local_guarantee is removed: nothing called it through the trait.
  • New hook, highest priority: AccuracyModel::accepts_output_regardless_of_guarantee(output, target) -> bool, default false. For each query output, Stage 3's accuracy check now works in this order:

    1. Executability first. A plan that reads Count-Min (plain Cms, CmsWithHeap or Hydra Count-Min) over update weights not proven non-negative is rejected with the new failed-check cause RejectionCause::CountMinWeightsNotProvenNonNegative (count_min_weights_not_proven_non_negative). It was AccuracyTargetMissed before.
      • Such a plan does not run: the executor refuses CmsWithHeap and HydraCms when it builds the plan, and plain Count-Min fails at the first negative weight. This check is the only place that rejects such a plan before execution, so the hook never overrides it.
      • The reason names the family, e.g. "q1: HydraCms needs non-negative update weights, … (samples of m, which is not declared a counter)". A Hydra structure is named by its Hydra kind, not "Cms".
    2. Ask the hook. If it returns true, the output is accepted whatever its guarantee says (missing, met or missed), and no guarantee or evidence check runs for it.
    3. Otherwise, run today's path unchanged: the Stage 1 guarantee, then the evidence's output_guarantee. If neither meets the target, the candidate is rejected:
      • as the new RejectionCause::NoAccuracyGuarantee (no_accuracy_guarantee) when neither source gives a guarantee;
      • as AccuracyTargetMissed when a guarantee misses.

    An output the hook accepts is recorded with AcceptedAccuracy::accepted_regardless_of_guarantee = true. Its summary reads "metric rank; accepted by the deployment's accuracy model regardless of guarantee (Kll)".

  • Behavior change (built-in models). The default hook accepts nothing, so selection is unchanged. Two causes change:

    • a candidate whose output has no guarantee from either source was accuracy_target_missed and is now no_accuracy_guarantee, with the reason "the output has no guarantee, and the deployment has no evidence for it, so it cannot meet target …";
    • a plan over unproven Count-Min weights was accuracy_target_missed and is now count_min_weights_not_proven_non_negative, with the same reason text.

    The updated tests show it:

    • count_min_over_signed_weights_is_rejected_as_invalid, count_min_over_samples_needs_a_declared_counter and combined_estimates_need_evidence_about_the_output (plan-selection)
    • combined_scalar_references_keep_summary_candidates (planner)
    • the regenerated dag-viewer fixture planner-layering-example1.json: 38 candidates change cause (all were accuracy_target_missed before), and each of the 19 accepted candidates gains accepted_regardless_of_guarantee: false on both queries. No candidate in that example reads Count-Min, so none moves to count_min_weights_not_proven_non_negative. P1 is still selected.
  • Devtools and viewer. stage_pipeline writes accepted_regardless_of_guarantee per accepted query. The dag-viewer's FAILED_CHECK_CAUSES includes count_min_weights_not_proven_non_negative and no_accuracy_guarantee. The viewer's Python test and the devtools fixture check read the failed-check set from the viewer and from RejectionCause::failed_a_check.

  • Removed (no non-test caller):

    • AccuracyModel::local_guarantee
    • DefaultAccuracyModel::sketch_guarantee
    • the duplicate re-exports accuracy::{local_guarantee, propagate} (use the module paths)
    • the crate-root re-export asap_logical_asap_optimization::PropagationStats (use accuracy::PropagationStats)

Closes #572

How

  • crates/plan-selection/src/accuracy.rs holds both traits and their defaults. BuiltinAccuracyModel::satisfies calls Stage 1's accuracy::satisfies. Stage 3's one composition (domain_bound) calls accuracy::composition::propagate directly. Dependencies stay one-way: plan-selection depends on logical-asap-optimization, and Stage 1 depends on nothing in plan-selection.
  • accuracy_check (plan-selection lib.rs) returns Err((RejectionCause, reason)). Per query it:
    1. collects the sketches read, and rejects a Count-Min over unproven weights as CountMinWeightsNotProvenNonNegative;
    2. asks the hook (accept, then continue to the next query);
    3. accepts if the theoretical guarantee meets the target, or else the evidence guarantee (evidence is read only when needed);
    4. otherwise rejects as NoAccuracyGuarantee when neither exists, else as AccuracyTargetMissed with today's reasons.
  • In Stage 1, the composition rules' private impl DefaultAccuracyModel { fn additive … } block becomes plain private functions. Only the indentation changes (git diff -w shows only the removed impl and Self:: lines). local_guarantee and propagate get the doc comments the trait methods used to carry.

Key code interfaces added / modified

crates/plan-selection/src/accuracy.rs (new module; re-exported from crates/plan-selection/src/lib.rs)

/// The deployment's accuracy decisions in plan selection (Stage 3): whether a guarantee
/// meets a target, and whether the deployment accepts an output directly. The trait stays;
/// it moved here from asap_logical_asap_optimization::accuracy, and a deployment passes it
/// in PlanningModels::accuracy. Guarantees compose with the built-in rules only.
pub trait AccuracyModel {
    // removed: fn propagate(&self, op, inputs, local, stats) -> Result<AccuracyGuarantee, AccuracyError>;
    //          Stage 3 called it only in domain_bound (ExactDivision with operand domains),
    //          which now calls accuracy::composition::propagate directly

    /// Whether `guarantee` meets a query's `target` (ε, δ, scope and metric); Stage 3
    /// rejects a candidate whose guarantee and evidence both fail it (AccuracyTargetMissed).
    fn satisfies(&self, guarantee: &AccuracyGuarantee, target: &AccuracyTarget) -> bool; // unchanged

    /// Whether the deployment accepts `output` (one query's output in a candidate) for
    /// `target` directly, whatever its guarantee says (missing, met or missed). Asked before
    /// any guarantee check; true skips every guarantee and evidence check for it. It does not
    /// override the Count-Min weight check before it (CountMinWeightsNotProvenNonNegative):
    /// the executor refuses CmsWithHeap and HydraCms at plan build, and plain Count-Min fails
    /// at the first negative weight. Default false.
    fn accepts_output_regardless_of_guarantee(
        &self,
        _output: &QueryRoot,
        _target: &AccuracyTarget,
    ) -> bool { false }                                                         // new

    // removed: fn local_guarantee(&self, family, query) -> Option<AccuracyGuarantee>;
    //          no caller through the trait; Stage 1 uses accuracy::estimators::local_guarantee
}

/// The planner's built-in rules as an AccuracyModel: Stage 1's conservative target check,
/// no output accepted regardless of its guarantee. The default
/// in PlanningModels::builtin(). The type stays; moved from the Stage 1 crate.
#[derive(Debug, Default, Clone, Copy)]
pub struct BuiltinAccuracyModel;                         // was DefaultAccuracyModel
// removed: DefaultAccuracyModel::sketch_guarantee(algorithm, params, query) (no caller)

/// The deployment's planning-time accuracy evidence, read only by Stage 3; passed in
/// PlanningModels::evidence. The trait stays; moved from asap_logical_asap_optimization::accuracy.
pub trait AccuracyEvidenceProvider {
    /// Enforced domain of `operand`'s true values (a quotient's or product's operands);
    /// None = unknown.
    fn operand_domain(&self, _operand: &OperatorNode) -> Option<OperandDomain> { None } // unchanged
    /// The deployment's empirical guarantee for one candidate's whole query output, checked
    /// with AccuracyModel::satisfies when the Stage 1 guarantee misses. None = no evidence.
    fn output_guarantee(&self, _output: &QueryRoot) -> Option<AccuracyGuarantee> { None } // unchanged
}

/// No deployment evidence; the default in PlanningModels::builtin(). The type stays; moved.
#[derive(Debug, Default, Clone, Copy)]
pub struct NoAccuracyEvidence;                            // moved

crates/plan-selection/src/lib.rs

/// Why Stage 3 did not select a candidate; JSON is the snake_case name. The type stays;
/// this PR adds one failed-check variant.
pub enum RejectionCause {
    /// Bad input for this candidate: its summaries, resized for the failure probabilities,
    /// do not rebuild into a plan.
    FailureProbabilityResizeFailed,   // unchanged
    /// Unsatisfiable: a query's output guarantee (and the evidence's) misses its target, and
    /// the model did not accept it directly.
    AccuracyTargetMissed,             // unchanged; no longer used for no guarantee or Count-Min weights
    /// Unsatisfiable: a query's plan reads Count-Min (Cms, CmsWithHeap or HydraCms) over update
    /// weights not proven non-negative (e.g. samples of a metric not declared a counter). It
    /// does not run, so the accuracy model's hook cannot accept it either.
    CountMinWeightsNotProvenNonNegative, // new (was reported as AccuracyTargetMissed)
    /// Unsatisfiable: a query's output has no guarantee (no Stage 1 rule, no evidence) and the
    /// deployment's model did not accept it directly.
    NoAccuracyGuarantee,              // new (was reported as AccuracyTargetMissed)
    /// Unsatisfiable: it scans raw data at query time and the deployment keeps none.
    RawDataNotRetained,               // unchanged
    /// Unsatisfiable: a query's latency is over its max_latency_ms bound.
    LatencyBoundExceeded,             // unchanged
    /// Unsatisfiable under cost model 3: its usage is over a cluster limit.
    OverClusterCapacity,              // unchanged
    /// Valid, lost: costs more than the selected candidate (cost models 1 and 2).
    Costlier,                         // unchanged
    /// Valid, lost: slower than the selected one (cost model 3, lowest-latency).
    Slower,                           // unchanged
    /// Valid, lost: more memory than the selected one (cost model 3, least-memory).
    MoreMemory,                       // unchanged
    /// Valid, lost: on the Pareto front but not its lowest-latency member.
    NotLowestLatencyOnParetoFront,    // unchanged
    /// Valid, lost: another candidate dominates it (pareto-front).
    Dominated,                        // unchanged
    /// Valid, lost: ties with the selected one and comes later in enumeration order.
    TiedLostOnEnumerationOrder,       // unchanged
}
/// Whether the candidate failed a check (now including CountMinWeightsNotProvenNonNegative
/// and NoAccuracyGuarantee) rather than
/// losing under the cost model.
pub fn failed_a_check(self) -> bool;  // unchanged signature; covers the new variant

/// The accuracy Stage 3 accepted for one query of a valid candidate. The type stays; this
/// PR adds a field.
pub struct AcceptedAccuracy {
    /// The target's metric; None when the user did not set one.
    pub target_metric: Option<AccuracyMetric>,          // unchanged
    /// The accepted guarantee's metric; None for an exact output or one accepted regardless
    /// of its guarantee.
    pub accepted_metric: Option<AccuracyMetric>,        // unchanged; doc covers the new case
    /// Whether the deployment's model accepted the output directly, whatever its guarantee
    /// (accepts_output_regardless_of_guarantee); false when a guarantee met the target.
    pub accepted_regardless_of_guarantee: bool,         // new
    /// The sketch algorithms the query's output reads, in plan order; empty when exact.
    pub sketches: Vec<String>,                          // unchanged
}

crates/plan-selection/src/pass/mod.rs

/// The deployment's models and settings plan selection reads. The type stays; the traits
/// behind `accuracy` and `evidence` moved to crate::accuracy.
pub struct PlanningModels<'a> {
    /// Decides whether a candidate's accuracy guarantee meets its query's target, and whether
    /// the deployment accepts an output regardless of its guarantee; BuiltinAccuracyModel by default.
    pub accuracy: &'a dyn AccuracyModel,                         // unchanged field; trait moved
    /// Operand domains and empirical output guarantees; default NoAccuracyEvidence.
    pub evidence: &'a dyn AccuracyEvidenceProvider,              // unchanged field; trait moved
    /// The cost model, resource speed and non-repeating query horizon; default settings.
    pub plan_selection: &'a PlanSelectionSettings,               // unchanged
    /// The summaries the executor supports and what it retains; default capabilities.
    pub deployment_capabilities: &'a DeploymentCapabilities,     // unchanged
}
/// These models with the deployment's accuracy model plugged into Stage 3.
pub fn with_accuracy(mut self, accuracy: &'a dyn AccuracyModel) -> Self;              // unchanged; trait moved
/// These models with the deployment's evidence.
pub fn with_evidence(mut self, evidence: &'a dyn AccuracyEvidenceProvider) -> Self;   // unchanged; trait moved

crates/logical-asap-optimization/src/accuracy/mod.rs, composition.rs, estimators/mod.rs, evidence.rs, lib.rs

// removed: pub trait AccuracyModel, pub struct DefaultAccuracyModel          (moved to plan-selection)
// removed: pub trait AccuracyEvidenceProvider, pub struct NoAccuracyEvidence (moved to plan-selection)
// removed from the crate-root re-exports: AccuracyModel, DefaultAccuracyModel,
//          AccuracyEvidenceProvider, NoAccuracyEvidence, PropagationStats (use accuracy::PropagationStats)

/// accuracy::estimators — the built-in guarantee of reading `query` from a summary of
/// `family` over exact input; None when the family has no error model (Sample, Wavelet,
/// StatModel, uncertified sketch evaluations): Pass 1 then keeps the candidate, guarantee None.
pub fn local_guarantee(family: &FieldDataType, query: &SketchStatistic)
    -> Option<AccuracyGuarantee>;                 // was pub(super); was DefaultAccuracyModel::local_guarantee

/// accuracy::composition — the built-in composition rule for `op`; Err = no rule or a
/// missing input (fail closed): Pass 1 keeps the candidate with guarantee = None.
pub fn propagate(
    op: &CompositionOperator,
    inputs: &[AccuracyGuarantee],
    local: Option<&AccuracyGuarantee>,
    stats: &PropagationStats,
) -> Result<AccuracyGuarantee, AccuracyError>;  // was pub(super); was DefaultAccuracyModel::propagate

/// accuracy — the built-in conservative check of a guarantee against a target (the body of
/// the old DefaultAccuracyModel::satisfies, unchanged); BuiltinAccuracyModel delegates to it.
pub fn satisfies(guarantee: &AccuracyGuarantee, target: &AccuracyTarget) -> bool; // new free fn

/// accuracy::evidence — an enforced value domain for one operand of an exact arithmetic op.
pub struct OperandDomain { /* lower, upper, max_samples, contract */ }     // unchanged, stays in Stage 1
/// accuracy::evidence — statistics a propagation rule may consult; all unknown in Stage 1.
pub struct PropagationStats { /* unchanged fields */ }                      // unchanged, stays in Stage 1

crates/devtools/src/bin/stage_pipeline.rs (document format asap-stage-pipeline/v1)

stage3_plan_selection.accepted[<id>][q].accepted_regardless_of_guarantee: bool  // new; true = the deployment's
                                                                                 // model accepted it directly
stage3_plan_selection.rejected[].cause: "no_accuracy_guarantee"                 // new failed-check cause
stage3_plan_selection.rejected[].cause: "count_min_weights_not_proven_non_negative" // new failed-check cause

Usage (end to end)

From crates/planner/tests/deployment_accuracy_model.rs. All three tests pass at the head SHA, and the ids, causes and summaries below come from running them.

// A deployment that has measured every output itself and accepts each one for any target.
// It implements the trait's two methods: `satisfies` (here: the built-in check) and the hook.
struct AcceptsEveryOutput;
impl asap_plan_selection::AccuracyModel for AcceptsEveryOutput {
    fn satisfies(&self, g: &AccuracyGuarantee, t: &AccuracyTarget) -> bool {
        BuiltinAccuracyModel.satisfies(g, t)
    }
    fn accepts_output_regardless_of_guarantee(&self, _output: &QueryRoot, _target: &AccuracyTarget) -> bool {
        true
    }
}

// (1) An output with NO guarantee. What the user writes:
// `quantile(0.99, rate(x[5m])) - quantile(0.5, rate(x[5m]))`, every 10 min, target
// AccuracyTarget::max_error(0.01).accuracy_metric(Rank), metric sampled every 15 s. Stage 1
// has no rule for a difference of rank estimates, so the summary candidates keep guarantee = None.
let workload =
    rank_target_workload("quantile(0.99, rate(x[5m])) - quantile(0.5, rate(x[5m]))");
// Built-in models: BuiltinAccuracyModel's hook says no; there is no root guarantee and no
// evidence. `builtin` is the PlanOutput (plans + selection report).
let builtin = e2e_plan(UserInput::new(&workload, FrontendInput::Promql { histograms: None },
                                      PlanningModels::builtin())).await?;
// `unguaranteed` = the 42 candidates rejected as NoAccuracyGuarantee (P6 … P47), reason
// "q1: the output has no guarantee, and the deployment has no evidence for it, so it cannot
// meet target …". The exact P5 is selected.
let unguaranteed = rejected_with(&builtin, RejectionCause::NoAccuracyGuarantee);
// The deployment's model, plugged into Stage 3 only.
let accepting_model = AcceptsEveryOutput;
// The same workload with it. Stage 3 asks the hook first for each of the 47 candidates' output
// and accepts every one. `accepting` is the PlanOutput to compare.
let accepting = e2e_plan(UserInput::new(&workload, FrontendInput::Promql { histograms: None },
    PlanningModels::builtin().with_accuracy(&accepting_model))).await?;
// Stages 1 and 2 produced the same candidates (ids, guarantees, sketch params) under both models.
assert_eq!(stage2_candidates(&workload, PlanningModels::builtin()),
           stage2_candidates(&workload, PlanningModels::builtin().with_accuracy(&accepting_model)));
// All 47 are accepted regardless of guarantee, e.g. "metric rank; accepted by the deployment's
// accuracy model regardless of guarantee (Kll)". Stage 3 selects P26, a plan that reads a KLL.
assert!(accepted_regardless_of_guarantee(&accepting).is_superset(&unguaranteed));

// (2) An output whose guarantee MISSES. What the user writes: `quantile_over_time(0.99, lat[5m])`,
// same target. Under the built-in models P3's relative bound 0.01 misses the rank target, so
// `missed` = {"P3"} (AccuracyTargetMissed).
let workload = rank_target_workload("quantile_over_time(0.99, lat[5m])");
// Under AcceptsEveryOutput, P3 is valid, priced, and accepted as "metric rank; accepted by the
// deployment's accuracy model regardless of guarantee (DDSketch)". So are P1 (exact) and P2
// (KLL), whose guarantees meet the target: every accepted output has
// accepted_regardless_of_guarantee = true. P1 is selected on cost.

// (3) What the hook cannot override (plan-selection test): Count-Min top-k over samples of `m`
// declared a Gauge. P2 (CmsWithHeap) is rejected as CountMinWeightsNotProvenNonNegative,
// "q1: CmsWithHeap needs non-negative update weights, … (samples of `m`, which is not declared
// a counter)", under AcceptsEveryOutput as under BuiltinAccuracyModel. The same plan with a
// Hydra Count-Min build reads "q1: HydraCms needs non-negative update weights, …".

The first test in the same file plugs in an ExactOnlyAccuracyModel (only satisfies, which accepts only exact guarantees; default hook) for quantile_over_time(0.99, lat[5m]). Stage 3 gets the same candidates P1–P3, with identical Stage 2 renderings. The KLL candidate P2, whose rank bound 0.00997 the built-in model accepts, is now rejected as AccuracyTargetMissed.

Before this PR

  • AccuracyModel, DefaultAccuracyModel and AccuracyEvidenceProvider lived in Stage 1's crate, though only Stage 3 read them. Stage 1 called its own rules as DefaultAccuracyModel.propagate(...).
  • A deployment model had to implement propagate, satisfies and local_guarantee, though Stage 3 called propagate only for one quotient case. It could only grade guarantees. An output with no guarantee could be accepted only through evidence, and a guarantee that missed could not be overridden.
  • An output with no guarantee was rejected as accuracy_target_missed, which reads like a bound that missed.

After this PR

  • The traits live in the only stage that uses them: asap_plan_selection::{AccuracyModel, BuiltinAccuracyModel, AccuracyEvidenceProvider, NoAccuracyEvidence}. Stage 1 exposes only its built-in rules, as free functions.
  • A deployment model implements satisfies, and may override accepts_output_regardless_of_guarantee, which accepts an output directly, ahead of every guarantee check: missing, met or missed. It never overrides the executor's requirement that Count-Min weights be proven non-negative. Guarantees compose with the built-in rules only.
  • A candidate rejected for having no guarantee, or for unproven Count-Min weights, says so (no_accuracy_guarantee, count_min_weights_not_proven_non_negative).
  • With the built-in models, every selected plan is the same as before. Only those two rejection causes and the no-guarantee reason change.

Migration for external code:

Before After
asap_logical_asap_optimization::{AccuracyModel, DefaultAccuracyModel} asap_plan_selection::{AccuracyModel, BuiltinAccuracyModel}
asap_logical_asap_optimization::{AccuracyEvidenceProvider, NoAccuracyEvidence} (or ::accuracy::…) asap_plan_selection::{AccuracyEvidenceProvider, NoAccuracyEvidence}
asap_logical_asap_optimization::PropagationStats asap_logical_asap_optimization::accuracy::PropagationStats
DefaultAccuracyModel.local_guarantee(f, q) asap_logical_asap_optimization::accuracy::estimators::local_guarantee(f, q)
DefaultAccuracyModel.propagate(...) in Stage 1 code asap_logical_asap_optimization::accuracy::composition::propagate(...)
DefaultAccuracyModel.satisfies(g, t) asap_logical_asap_optimization::accuracy::satisfies(g, t) or BuiltinAccuracyModel.satisfies(g, t)
impl AccuracyModel { fn local_guarantee …; fn propagate …; fn satisfies … } impl AccuracyModel { fn satisfies … } (accepts_output_regardless_of_guarantee optional; propagate and local_guarantee removed)
models.accuracy.propagate(op, inputs, local, stats) asap_logical_asap_optimization::accuracy::composition::propagate(op, inputs, local, stats)
cause accuracy_target_missed for an output with no guarantee no_accuracy_guarantee
cause accuracy_target_missed for unproven Count-Min weights count_min_weights_not_proven_non_negative
AcceptedAccuracy { target_metric, accepted_metric, sketches } + accepted_regardless_of_guarantee: bool

Limitations and follow-ups:

  • The hook sees one query's output and its target only, not the workload entry or data workload. A deployment that needs more context can capture it in its model value.
  • An accepted-by-hook output records no metric (accepted_metric: None), because no guarantee was checked.
  • These Markdown docs still describe the old layout. Doc updates are tracked in Update the docs after the G1–G11 consolidation groups land #677, so this PR does not edit them:
    • docs/develop_docs/asap-aware-mapping-architecture.md:69
    • docs/develop_docs/asap-aware-mapping-contracts.md:64
    • docs/develop_docs/end-to-end-accuracy-guarantees.md:56
    • docs/develop_docs/extend-asap-aware-mapping.md:26
    • docs/develop_docs/library-api.md:180 and docs/design_docs/architecture/input-output-workflow.md:21,235,242 (DefaultAccuracyModel name)
    • docs/design_docs/concepts/accuracy-models.md:143 and :287 (the Stage 1 accuracy module doc points readers there)
    • docs/develop_docs/asap-aware-mapping-contracts.md:98 (its RejectionCause variant list lacks count_min_weights_not_proven_non_negative and no_accuracy_guarantee)
    • docs/user_guide_docs/run-a-query.md:39-43: P2 is now count_min_weights_not_proven_non_negative and P3 is no_accuracy_guarantee, not accuracy_target_missed
    • tools/dag-viewer/README.md (the cause list lacks count_min_weights_not_proven_non_negative and no_accuracy_guarantee)

Verification

All runs below are on the head, rebased on main b433a3e. After the rebase I regenerated the fixture with stage_pipeline --example planner-layering-1. It is unchanged from the merged result, and the counts above are recounted against b433a3e's fixture.

  • cargo nextest run -p asap-planner --test deployment_accuracy_model: 3 passed.
    • …_accepts_outputs_without_a_guarantee: the hook accepts outputs with no guarantee; the default model rejects them.
    • …_accepts_outputs_whose_guarantee_misses: the hook accepts P3, which the built-in model rejects as AccuracyTargetMissed. Every accepted output, including P1 and P2 whose guarantees meet the target, has accepted_regardless_of_guarantee == true. Moving the hook after the met-guarantee acceptance makes this test fail; I reverted the move.
    • …_decides_only_in_plan_selection: the default hook leaves today's path unchanged.
    • The hook tests first failed to compile, with no hook, no NoAccuracyGuarantee and no accepted_* field.
  • plan-selection unproven_count_min_weights_reject_even_an_accepting_model (written first; it failed to compile with no such cause): with m declared a Gauge, P2 (CmsWithHeap) is rejected as CountMinWeightsNotProvenNonNegative, with "not declared a counter", under an always-accepting hook and under the built-in model. Passes.
  • plan-selection unproven_hydra_count_min_weights_name_hydra_cms: P2 with its CmsWithHeap build replaced by a Hydra Count-Min, over a Gauge. The reason says "HydraCms needs non-negative update weights" and "not declared a counter". Written first; it failed with "Cms needs …" before the fix.
  • plan-selection the_accuracy_model_accepts_an_output_whatever_its_guarantee: the summary candidates of a quantile difference are rejected by the built-in model, as AccuracyTargetMissed with evidence that misses and as NoAccuracyGuarantee with no evidence. An accepting hook makes them valid in both cases. Passes.
  • cargo nextest run over asap-plan-selection (whole crate), asap-planner (whole crate), devtools stage_pipeline, frontend-promql univmon_candidates, integration-tests promql_to_post_asap, and logical-asap-optimization accuracy:: / per_computation_candidates::: 340 passed.
    • When the no-guarantee cause landed, five failed. I updated two plan-selection tests (count_min_over_samples_needs_a_declared_counter, combined_estimates_need_evidence_about_the_output), one planner test (combined_scalar_references_keep_summary_candidates) and the devtools fixture check (fixture regenerated with stage_pipeline --example planner-layering-1).
    • The fifth, a_bound_in_another_metric_misses_the_target, passed again once the evidence-miss reason kept its old wording.
    • When the Count-Min cause landed, one more failed and was updated: count_min_over_signed_weights_is_rejected_as_invalid (plan-selection).
  • python3 -m unittest test_viewer (tools/dag-viewer): 37 passed.
  • cargo fmt --all -- --check: clean.
  • cargo clippy --all-targets --all-features --locked -- -D warnings on logical-asap-optimization, plan-selection, frontend-promql, planner, integration-tests and devtools: clean.
  • I did not run the full workspace suite locally (repo policy). GitHub CI runs it.

@zzylol zzylol added accuracy Accuracy requirements, guarantees, and propagation extensibility optimizer Workload optimization and plan selection labels Oct 18, 2026
@zzylol
zzylol force-pushed the refactor/accuracy-model-plan-selection branch from 893964d to af12b84 Compare October 18, 2026 15:31
@zzylol zzylol changed the title refactor!: move AccuracyModel to plan-selection (Stage 3) refactor!: move AccuracyModel and evidence to plan-selection; let the model decide outputs without a guarantee Oct 18, 2026
@zzylol
zzylol force-pushed the refactor/accuracy-model-plan-selection branch 2 times, most recently from fedc920 to f8c3fbd Compare October 18, 2026 16:10
@zzylol zzylol changed the title refactor!: move AccuracyModel and evidence to plan-selection; let the model decide outputs without a guarantee refactor!: move AccuracyModel and evidence to plan-selection; let the model accept outputs regardless of guarantee Oct 18, 2026
@zzylol
zzylol force-pushed the refactor/accuracy-model-plan-selection branch from 8163234 to 2106009 Compare October 18, 2026 16:48
Stage 3 is the only stage that uses the deployment's accuracy model
(#509, #572 Q22). The AccuracyModel trait and DefaultAccuracyModel move
from asap-logical-asap-optimization to asap_plan_selection::accuracy.
Stage 1 calls the built-in analytical rules directly: the free functions
accuracy::{local_guarantee, propagate, satisfies}.

BREAKING CHANGE: asap_logical_asap_optimization::{AccuracyModel,
DefaultAccuracyModel} are removed; use asap_plan_selection::{AccuracyModel,
DefaultAccuracyModel}. AccuracyModel::local_guarantee and
DefaultAccuracyModel::sketch_guarantee are removed (no caller).
…puts without a guarantee

- Move AccuracyEvidenceProvider and NoAccuracyEvidence to
  asap_plan_selection::accuracy, next to AccuracyModel; OperandDomain and
  PropagationStats stay in Stage 1.
- Rename DefaultAccuracyModel to BuiltinAccuracyModel. AccuracyModel::propagate
  gets a default body (the built-in rules), so a deployment model implements
  only satisfies.
- Add AccuracyModel::accepts_output_without_guarantee (default false). Stage 3
  asks it only when neither Stage 1 nor the evidence supplies a guarantee;
  an output it accepts is recorded with AcceptedAccuracy::accepted_without_guarantee,
  and one it does not is rejected with the new RejectionCause::NoAccuracyGuarantee
  (was AccuracyTargetMissed).
- Document accuracy::estimators::local_guarantee and
  accuracy::composition::propagate, and drop their duplicate re-exports.
- stage_pipeline writes accepted_without_guarantee; the dag-viewer accepts
  the new cause; the example fixture is regenerated.

BREAKING CHANGE: AccuracyEvidenceProvider/NoAccuracyEvidence move to
asap_plan_selection; DefaultAccuracyModel is now BuiltinAccuracyModel; a
candidate whose output has no guarantee is rejected as no_accuracy_guarantee
instead of accuracy_target_missed; AcceptedAccuracy gains a field.
…output regardless of its guarantee

The hook now has priority over every guarantee: Stage 3 asks
AccuracyModel::accepts_output_regardless_of_guarantee first for each query
output, and an output it accepts skips the guarantee, Count-Min weight and
evidence checks. Otherwise the check is unchanged (Stage 1 guarantee, then
evidence, then AccuracyTargetMissed, or NoAccuracyGuarantee when neither
gives one). The default returns false, so the built-in models behave as
before. Tests pin that the hook accepts an output whose guarantee misses
and one with no guarantee, in Stage 3 and end to end.

Also drop the unused crate-root re-export of PropagationStats and
re-indent the accuracy rejection reasons.

BREAKING CHANGE: AccuracyModel::accepts_output_without_guarantee is now
accepts_output_regardless_of_guarantee, and
AcceptedAccuracy::accepted_without_guarantee (stage_pipeline JSON key
accepted_without_guarantee) is now accepted_regardless_of_guarantee;
asap_logical_asap_optimization::PropagationStats is now only at
accuracy::PropagationStats.
…curacy model's hook

The executor cannot run Count-Min (CmsWithHeap, Hydra Count-Min) over
update weights not proven non-negative, and Stage 3's weight check is the
only place that rejects such a plan. accepts_output_regardless_of_guarantee
skipped it. The check now runs before the hook and rejects with the new
failed-check cause RejectionCause::CountMinWeightsNotProvenNonNegative
(count_min_weights_not_proven_non_negative); the hook still overrides
every guarantee. A test also pins that the hook overrides guarantees that
meet the target.

BREAKING CHANGE: a plan rejected for unproven Count-Min weights now has
cause count_min_weights_not_proven_non_negative instead of
accuracy_target_missed.
@zzylol
zzylol force-pushed the refactor/accuracy-model-plan-selection branch from 2106009 to f442f0b Compare October 18, 2026 17:18
…Count-Min in the weights rejection

Stage 3 composed through the model only for a ratio of relative
estimates with enforced operand domains (domain_bound, ExactDivision);
every other guarantee comes from Stage 1's built-in rules. The trait now
has satisfies and accepts_output_regardless_of_guarantee, and
domain_bound calls accuracy::composition::propagate directly.

A Hydra (SharedMultiSubpopulation) summary is now named by its Hydra kind,
so the unproven-weights reason reads "HydraCms needs non-negative update
weights" rather than "Cms". Document which Count-Min forms the weight
check covers and why: the executor refuses CmsWithHeap and HydraCms at
plan build, and plain Count-Min fails at the first negative weight.

BREAKING CHANGE: AccuracyModel::propagate is removed; a deployment model
cannot change how guarantees compose.
@zzylol
zzylol force-pushed the refactor/accuracy-model-plan-selection branch from f442f0b to c0bb156 Compare October 18, 2026 17:27
@zzylol
zzylol merged commit 6d9f799 into main Oct 18, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

accuracy Accuracy requirements, guarantees, and propagation extensibility optimizer Workload optimization and plan selection

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reorganize crates and modules by #509 stages and the #511 unified IR

1 participant