Repository navigation
feat(types): three error metrics, a shared AccuracySpec and metric-aware satisfies (#690) - #708
Merged
Merged
Conversation
zzylol
force-pushed
the
feat/accuracy-spec-690
branch
from
October 15, 2026 14:21
4ed75f4 to
6e4768e
Compare
…targets (#690) ErrorMetric shrinks to Absolute, Relative and Rank. Absolute carries an optional norm scale (StreamL1 for Count-Min, StreamL2 for Count Sketch); Cardinality becomes Relative and TopKMembership is removed. AccuracySpec<E, P> {metric, error, failure_probability, scope} is the outer shape of both a target and a guarantee. AccuracyTarget is AccuracySpec<f64, Option<f64>> (no δ requirement when None), with exact()/epsilon()/epsilon_delta() constructors. AccuracyGuarantee holds a symbolic AccuracySpec<BoundExpr, ProbabilityExpr> plus its provenance. AccuracyScope is PerValue or WholeResult; guarantees are PerValue and targets default to WholeResult. Workloads and the query-language frontends take a RequestedAccuracy whose metric may be omitted. It still reads the old "Exact"/Epsilon/EpsilonDelta JSON. Plan selection resolves it per query root: counts and top-k default to an L1-scaled absolute error, distinct counts to a relative error, and any other root to an absolute value error. A quantile or histogram_quantile root without a metric is an input error. Example and test workloads with quantile targets now name Rank. A summary is still checked against its intent's target in its own native terms. stage_pipeline gains --metric. The committed planner-layering-example1 viewer fixture is regenerated for the new JSON shape; its selection is unchanged.
DefaultAccuracyModel::satisfies now compares a guarantee with a target only when both are stated in the same metric. An L2-scaled bound also meets an L1-scaled target of the same stream, since ‖f‖₂ ≤ ‖f‖₁. A bound in any other metric does not satisfy, however small, and an exact guarantee meets every target. This takes over the separate answers check proposed in #627. A target without a δ checks only the error. A per-value guarantee is still compared with a whole-result target directly until the union bound in #669b; a test pins this behavior. Plan selection's rejection reason now names the guarantee's metric. Example 2's Q3 (an L2 norm, whose root defaults to an absolute error) now names a relative target, the metric UnivMon certifies its L2 in.
The editor gains a metric dropdown next to ε (default, Absolute, Relative, Rank). /api/plan passes a non-empty metric to `stage_pipeline --metric` and rejects an unknown metric or a metric without ε. The query header shows the metric a target names. A quantile query planned with a metric selects a plan; without one, the page shows the missing-metric error.
zzylol
force-pushed
the
feat/accuracy-spec-690
branch
from
October 15, 2026 14:35
6e4768e to
50256ca
Compare
This was referenced Oct 15, 2026
This was referenced Oct 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #690. Supersedes #627: its separate
answerscheck is folded into the metric-awaresatisfies.Why
AccuracyTarget(Exact | Epsilon | EpsilonDelta) had no error metric, andsatisfiescompared only the number. Each summary states ε in its own form: KLL as a rank error, DDSketch as a relative error, Count-Min as a fraction of the stream's L1 norm, Count Sketch of its L2 norm, HLL as a relative distinct-count error. All of these were compared against the same ε. A bound in one metric could pass a target that meant another. On the guarantee side,ErrorMetrichad seven variants, and most were the same error under a different normalization.What
Three error metrics.
ErrorMetricis nowAbsolute { scale },RelativeandRank, and stays#[non_exhaustive].scaleisNone(value units),StreamL1orStreamL2.Absolute@StreamL1and Count SketchAbsolute@StreamL2.Relative.TopKMembershipis removed.One outer shape.
AccuracySpec<E, P> { metric, error, failure_probability, scope }is shared by both sides:AccuracyTarget = AccuracySpec<f64, Option<f64>>. δNonemeans no δ requirement; sizing then uses the default δ. Constructors areexact(),epsilon(ε)andepsilon_delta(ε, δ), pluswith_metricandwith_scope.AccuracyGuarantee { spec: AccuracySpec<BoundExpr, ProbabilityExpr>, provenance }.provenanceis the explanation trail, and no planning decision reads it.AccuracyScopeisPerValue | WholeResult. Guarantees arePerValue, and targets default toWholeResult.Unresolved input vs concrete target. Workloads and the query-language frontends take a
RequestedAccuracy, whose metric may be omitted.It still reads the old
"Exact",{"Epsilon": ε}and{"EpsilonDelta": …}JSON, as well as{"metric", "error", "failure_probability", "scope"}.Plan selection resolves it per query root (
RequestedAccuracy::for_root), and the metric applies only at the root:Absolute@StreamL1Relativequantile,histogram_quantileSelectionError::AccuracyRequest)AbsoluteMetric-aware
satisfies.‖f‖₂ ≤ ‖f‖₁over the root's one stream (stated inmetric_meetsandNormScale).Sizing stays native (§4). A summary is checked against its intent's target restated in the summary's own metric (
accuracy::native_target). Only the root target's metric is checked, by Stage 3.stage_pipelinegains--metric <absolute|absolute_stream_l1|absolute_stream_l2|relative|rank>.Viewer. The dag-viewer editor has a metric dropdown next to ε (default, Absolute, Relative, Rank).
/api/planpasses a non-empty metric tostage_pipeline --metricand rejects an unknown metric or a metric without ε. The query header shows the metric a target names.How
crates/types/src/types.rs:ErrorMetric,NormScale,AccuracyScope,AccuracySpec,RequestedAccuracy, the root-metric derivation andAccuracyRequestError.post_asap::guaranteere-exports them, andAccuracyGuarantee::newbuilds a per-value guarantee.pass/stage1_to_stage3.rs:resolve_targetsturns each query'sRequestedAccuracyinto the root'sAccuracyTargetbeforeselect_plan.accuracy/mod.rs:satisfieswithmetric_meets, andnative_target. Pass 1's per-summary checks (replacement.rs,grouping.rs) usenative_target..bound→.spec.error,AccuracyTarget::Epsilon(e)→epsilon(e), …).Rankon KLL paths (Examples 3 and 4, the e2e and summary-sharing tests), andRelativewhere the test checks a relative error (DDSketch paths, the tumbling-pane p99). Example 2's Q3 (an L2 norm) uses the newRelativedefault, the metric UnivMon certifies its L2 in.tools/dag-viewer/{index.html,editor.js,app.js,server.py}: the metric dropdown, its pass-through and its display.Key code interfaces added / modified
crates/types/src/types.rscrates/types/src/post_asap/guarantee.rs,workload.rscrates/asap-aware-mapping/src/accuracy/mod.rs,stage3_plan_selection.rs,pass/stage1_to_stage3.rscrates/frontend-metricsql/src/lib.rs,frontend-sql/src/lib.rs,frontend-promql/src/promql.rscrates/integration-tests/src/lib.rs,crates/devtools/src/lib.rs,planning_stages.rs,bin/stage_pipeline.rs,crates/asap-physical-operators/tests/common/mod.rstools/dag-viewer/server.py,editor.js,index.htmlUsage, end to end (numbers from
stage_pipelineat this PR's merge commitac3c852a):The same request in the workload JSON:
"requirements": {"accuracy": {"metric": "rank", "error": 0.01, "failure_probability": 0.001}}(scope defaults to whole result; the old{"EpsilonDelta": {"epsilon": 0.01, "delta": 0.001}}still parses, with no metric).Before this PR
satisfies(g, t)checked onlyg.bound ≤ ε(and δ). A DDSketch relative bound of 0.01 met a rank target of 0.01. A Count-Min L1 frequency bound met a distinct-count target whenever its number was small enough. A quantile target needed no metric.After this PR
(The DDSketch candidate P3 is rejected for a rank target; KLL P2 is selected.) The committed
planner-layering-example1viewer fixture is regenerated for the new JSON shape, and its selection is unchanged (P1; P4–P9 rejected for q2).Verification
accuracy::tests::satisfies_requires_the_targets_metric(every metric pair).a_target_without_delta_ignores_the_failure_probability.per_value_guarantees_meet_whole_result_targets_directly_until_669b, which pins the current scope behavior.stage3_plan_selection::tests::a_bound_in_another_metric_misses_the_target.types::tests::{requested_accuracy_reads_old_and_spec_forms, metric_wire_names, root_metric_defaults_follow_the_query_intent, a_quantile_root_must_name_its_metric, exact_targets}.root_metric_defaults_follow_the_query_intentcovers L2 →Relativeand entropy →Absolute.stage_pipeline::a_quantile_target_names_its_metric.test_server.test_metric_names_the_targets_error_metric, the new metric validation cases,test_server.PlanEndpointTests.test_a_quantile_query_plans_with_the_metric_it_names(plans a quantile query through/api/planwith and without a metric), andtest_viewer.test_the_editor_names_a_target_metric.satisfies_requires_the_targets_metric,a_bound_in_another_metric_misses_the_target) fail with the metric check disabled.Absolute { scale: None }vsRelative).cargo fmt --all -- --checkpasses.cargo clippy --workspace --all-targets --all-features --locked -- -D warningsis clean.cargo nextest runover the whole workspace, in two runs: 535 + 1188 passed, 0 failed.python3 -m unittest test_viewer test_serverintools/dag-viewer: 48 passed.Limitations and follow-ups
PerValueguarantee is checked against aWholeResulttarget directly. The G·δ union bound and budget allocation are #669b.OperandDomainevidence is not implemented insatisfies. Such pairs are unmet unless a composition rule already produced the target's metric.AccuracyTargetwith a metric (used only for sizing, in native terms). A quantile intent without a named metric is sized asRank.