Skip to content

refactor!: arrange crates by planning stage - #741

Merged
zzylol merged 2 commits into
mainfrom
refactor/stage-crates
Oct 18, 2026
Merged

zzylol merged 2 commits into
mainfrom
refactor/stage-crates

Conversation

@zzylol

@zzylol zzylol commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

Why

The code layout still followed the pre-#509 split. asap-types had pre_asap / post_asap modules whose names no longer described their contents, and asap-aware-mapping held Stages 1, 2 and 3 in one crate, so nothing stopped Stage 1 from importing Stage 3 code. #572 (approved 2026-10-03, with the crate names decided 2026-10-13) arranges the crates by planning stage, so that Cargo enforces the one-way stage flow, and arranges asap-types by #511 IR section.

What

This is a pure move: file paths, use lines, Cargo.toml files and the docs that name crate paths change. No planning logic changes, and selections and costs are unchanged.

  • asap-types keeps its name. pre_asap, post_asap, types, resources and parsed_workload are gone. Their contents now live under ir::{operator, scalar, schema, properties}, workload and physical.
  • asap-aware-mapping is split into three stage crates: asap-logical-asap-optimization (Stage 1), asap-physical-asap-optimization (Stage 2) and asap-plan-selection (Stage 3).
  • asap-physical-operators is renamed asap-executor (Stage 4). No stage crate depends on it.
  • The asap-planner facade re-exports from asap-plan-selection, as it did before from asap-aware-mapping::pass.

How

The moves were done with a script:

  1. git mv every file.
  2. A path rewriter resolves each use tree and inline path (crate::, super::, extern) against the file's old module and maps it to its new module. It keeps the text as written when the path still resolves, and rewrites the path otherwise.
  3. The module roots (lib.rs, the new mod.rs files) were written by hand.
  4. Items that a later crate now reads were made pub (list below).
  5. Docs and comments that name crate paths were rewritten the same way, then checked by hand.

Cycles and how each was resolved:

  • The Stage 1 to Stage 3 driver (pass/) runs all three stages, so it goes into plan-selection with PlanningModels, not into the facade. Stage 3's unit tests call run_stage1_to_stage3, and they could not do that from a crate below the facade. The facade re-exports it.
  • Stage 1 enumeration (stage1_logical_asap_optimization, Stage1Candidate, Pass2Candidate, and the private Pass 2 driver Pass2/Rules/Pass2List/pass2_candidates/next_choice) moved out of pass/stage1_to_stage3.rs into Stage 1's new workload_candidates.rs, together with the test that covers them. It called Stage 2's fixed_interval_ms, a pure function of QueryRecurrence, so that function moved with it, and Stage 2 now imports it from Stage 1.
  • Shared test helpers. test_support is used by the tests of all three stage crates. It now lives in Stage 1 as pub mod test_support behind the test-support feature. The later stage crates enable that feature in their dev-dependencies. cms_summary used to run the whole pipeline only to read its Stage 1 candidates. It now calls stage1_logical_asap_optimization on the same pruned Pass 1 inventory, with the same built-in deployment and recurrence, so it returns the same node.
  • ExecutionTiming and Materialization were in one file. Medium and Materialization (the Stage 2 decision) moved to types::physical::materialization. ExecutionTiming, ExecutionDataState, DataPrimitive and ExecutionDataStateError moved to types::ir::properties::execution_data_state, as Reorganize crates and modules by #509 stages and the #511 unified IR #572 recorded.

Old path → new crate / module

Old New
asap_types::ir::{node, non_asap, asap, operator_properties, query, pane_chain} asap_types::ir::operator::* (the ir:: re-exports OperatorNode, ASAPOp, QueryRoot, … stay)
asap_types::pre_asap::agg_intent, post_asap::maintained_population asap_types::ir::operator::{agg_intent, maintained_population}
asap_types::pre_asap::{expr_ir, scalar_type_rules, column_resolution} asap_types::ir::scalar::*
asap_types::pre_asap::schema asap_types::ir::schema
asap_types::post_asap::{sketch, family_capability}, ir::{aggregate_schema, error, schema_support} asap_types::ir::schema::*
asap_types::types (AccuracyTarget, AccuracyMetric, …) asap_types::ir::properties::accuracy (re-exported from ir::properties)
asap_types::post_asap::guarantee, ir::{timing, summary_coverage} asap_types::ir::properties::*
asap_types::post_asap::execution_data_state split: ir::properties::execution_data_state and physical::materialization (Materialization, Medium)
asap_types::post_asap::summary_window asap_types::physical::summary_window
asap_types::post_asap::query_time (posterior error-bound math, no callers) asap_executor::query_time
asap_types::{workload, parsed_workload, resources} asap_types::workload, workload::parsed_workload, workload::resources
pre_asap::X / post_asap::X re-exports the section that defines X: ir::operator::X, ir::scalar::X, ir::schema::X, ir::properties::X, physical::X
asap_aware_mapping::{per_computation_candidates, asap_aware_cse, realization, accuracy, deployment_capabilities, maintained_population, function_rules, test_support} asap_logical_asap_optimization::*
asap_aware_mapping::pass::{stage1_logical_asap_optimization, Stage1Candidate, Pass2Candidate} asap_logical_asap_optimization::workload_candidates::*
asap_aware_mapping::stage2_physical_asap_optimization asap_physical_asap_optimization (crate root)
asap_aware_mapping::stage3_plan_selection (+ measured_operation_costs) asap_plan_selection (crate root)
asap_aware_mapping::{analytical_cost, physical_operator_statistics, physical_handoff_cost, query_physical_lowering, storage_io} asap_plan_selection::cost::*
asap_aware_mapping::pane_sharing asap_plan_selection::pane_sharing
asap_aware_mapping::pass (optimize, OptimizationPass, PassRegistry, PlanningModels, run_stage1_to_stage3, …) and its root re-exports asap_plan_selection::pass and asap_plan_selection::*
asap_aware_mapping::{AccuracyModel, DeploymentCapabilities, summary_candidates, …} root re-exports asap_logical_asap_optimization::*
crates/asap-aware-mapping/tests/per_computation_candidates.rs crates/logical-asap-optimization/tests/
crates/asap-aware-mapping/tests/{physical_handoff_cost, storage_io}.rs crates/plan-selection/tests/
crates/asap-physical-operators (asap_physical_operators) crates/executor (asap_executor)

Crate dependency graph (normal dependencies)

asap-types ◄── asap-frontend-{common, sql, promql, metricsql}          Stage 0
asap-types ◄── asap-logical-asap-optimization                          Stage 1
asap-types, Stage 1 ◄── asap-physical-asap-optimization                Stage 2
asap-types, Stage 1, Stage 2 ◄── asap-plan-selection                   Stage 3 (+ PlanningModels, Stage 1→3 driver)
asap-plan-selection, frontends ◄── asap-planner                        facade
asap-types ◄── asap-executor                                           Stage 4 (no stage crate depends on it)
asap-devtools, asap-integration-tests ◄── all of the above

Dev-dependencies only: the frontends' tests and the executor's tests use Stage 1 and plan-selection. Stages 2 and 3 enable asap-logical-asap-optimization/test-support. The facade's tests use Stage 1.

Visibility changes (pub(crate) → pub), needed by a later stage crate

  • Stage 1 accuracy::composition: INPUT_ROW_COUNT, DIVISION_OPERAND_DOMAINS, PRODUCT_OPERAND_DOMAINS, STREAM_L1_NORM, samples_sign_statistic, parse_samples_sign_statistic, parse_source_norm_statistic
  • Stage 1 accuracy::estimators: with_failure_probability; hll::{classic_relative_error, classic_failure_statistic, parse_classic_failure_statistic, classic_failure_probability, min_precision}; univmon::{parse_incomplete_top_layer_statistic, incomplete_top_layer_probability}
  • Stage 1 per_computation_candidates: approximate_inputs, root_guarantee, with_resized_summaries, distinct_bound_source, project_column_guarantee, operand_of, operand_guarantee, operand_domain_statistic
  • Stage 1 realization::ranked_column
  • Stage 1 workload_candidates::fixed_interval_ms (was pub(crate) in Stage 2)
  • every test_support helper (feature-gated, test-only)

Key code interfaces added / modified

Module paths only. No type, field, function signature or wire format changes. The module paths are listed in the table above. Crates added, renamed or removed:

# crates/logical-asap-optimization/Cargo.toml (new; was part of asap-aware-mapping)
[package]
name = "asap-logical-asap-optimization"   # Stage 1: Pass 1, Pass 2, analytical accuracy model
[features]
test-support = ["dep:asap-frontend-promql"]  # new: the shared test helpers (`test_support`)

# crates/physical-asap-optimization/Cargo.toml (new; was asap-aware-mapping::stage2_physical_asap_optimization)
name = "asap-physical-asap-optimization"

# crates/plan-selection/Cargo.toml (new; was asap-aware-mapping::{stage3_plan_selection, cost modules, pass})
name = "asap-plan-selection"

# crates/executor/Cargo.toml (was crates/asap-physical-operators)
name = "asap-executor"                     # was asap-physical-operators

# crates/asap-aware-mapping: removed

Usage: a downstream crate that planned through asap_aware_mapping now depends on asap-plan-selection. For example, tools/verify_metricsql_external_consumer.sh lowers MetricsQL with asap_types::ir::properties::AccuracyTarget::exact(), which was asap_types::types::AccuracyTarget::exact(). asap-planner's API is unchanged.

Deviations from #572

  • Crate names. The crates take the stage names decided on 2026-10-13 (logical-asap-optimization, physical-asap-optimization, plan-selection), not Reorganize crates and modules by #509 stages and the #511 unified IR #572's logical-optimizer / physical-optimizer. Inside Stage 1, Pass 1 and Pass 2 keep their module names, per_computation_candidates and asap_aware_cse, rather than pass1/ and pass2/.
  • Driver placement. The pass/ driver and PlanningModels are in plan-selection, not in the facade, for the reasons under How. The facade re-exports them.
  • AccuracyModel stays in Stage 1. Pass 1 still calls DefaultAccuracyModel through the AccuracyModel trait (eight calls in per_computation_candidates.rs). Moving the trait to plan-selection needs those calls changed to the inherent estimator functions, which is a code change, not a move. The trait and AccuracyEvidenceProvider stay in asap_logical_asap_optimization::accuracy, and plan selection uses them through PlanningModels as before. That change is left for a follow-up PR.
  • Empirical accuracy. There is no plan-selection/accuracy/. erp and empirical_comparison were already deleted before this PR.
  • Modules not deleted. pane_sharing (to plan-selection) and query_time (to the executor) are moved, not deleted. Deleting them belongs to the PRs that replace them.
  • Leftover pre-docs: propose workload-wide planning, summary sharing, and materialization #509 names. These are not renamed here, because renaming them changes a wire key or a fixture: the pre_asap_sub_dag export key, tools/dag-viewer/post_asap_fixture.json, and the integration-test helper module asap_integration_tests::post_asap.
  • Deleted test. asap-aware-mapping/tests/cost_independence.rs scanned source files to check that Stage 1 and Stage 2 import no cost module. Cargo now enforces this: neither crate depends on asap-plan-selection. Its 2 tests are the only ones removed (1803 → 1801 #[test]s).

Part of #572. It does not close #572: the AccuracyModel move and the leftover names above remain.

Before this PR

All of Stages 1–3 lived in asap-aware-mapping, and stage boundaries were held only by a source-scanning test. asap-types exposed pre_asap, post_asap and types modules that cut across #511's sections.

After this PR

Each stage is its own crate. A Stage 1 or Stage 2 change that imports Stage 3, or any stage that imports the executor, fails to build. asap-types is arranged as ir::{operator, scalar, schema, properties}, workload and physical. The architecture doc, the extension guide and the library API guide name the new crates and paths.

Verification

Run locally at the PR head:

  • cargo check --workspace --all-targets: clean, no warnings.
  • cargo fmt --all -- --check: clean.
  • cargo clippy -p asap-types -p asap-logical-asap-optimization -p asap-physical-asap-optimization -p asap-plan-selection -p asap-executor -p asap-planner --all-targets --all-features --locked -- -D warnings: clean.
  • cargo nextest run -p asap-types -p asap-logical-asap-optimization -p asap-physical-asap-optimization -p asap-plan-selection -p asap-executor -p asap-planner -p asap-devtools: 901 passed, 7 skipped.
  • The example tests that pin selections and costs, plus the end-to-end stage tests: cargo nextest run -p asap-integration-tests -E 'binary(/planner_layering/) | binary(reuse_rules) | binary(stage1_logical_asap_optimization) | binary(materialization_e2e) | binary(stage2_physical_asap_optimization_sql) | binary(deployment_capabilities) | binary(promql_to_post_asap)': 129 passed, 21 skipped (#[ignore] tests).
  • cargo test --doc for the moved crates (the executor README doctest): passed.
  • Test count: 1803 → 1801 #[test]s, which are the 2 deleted cost_independence tests.

tools/verify_metricsql_external_consumer.sh was updated to the new path but not run locally. CI runs it.

asap-types is arranged by IR section: ir::{operator, scalar, schema,
properties}, workload and physical replace pre_asap, post_asap, types,
resources and parsed_workload. Materialization and Medium move to
physical::materialization; ExecutionTiming and the data-state types to
ir::properties::execution_data_state. The unused query-time error
estimation moves to the executor.

asap-aware-mapping is split into one crate per stage:
asap-logical-asap-optimization (Stage 1: Pass 1, Pass 2, whole-workload
candidate enumeration, analytical accuracy), asap-physical-asap-optimization
(Stage 2) and asap-plan-selection (Stage 3, its cost modules, PlanningModels
and the Stage 1 to Stage 3 driver). asap-physical-operators is renamed
asap-executor. Cargo now enforces the one-way stage flow, so the
source-scanning cost_independence test is removed.

Paths, use lines and manifests only. Items a later stage reads become pub,
the shared test helpers sit behind Stage 1's test-support feature, and
cms_summary reads Stage 1 candidates directly instead of running the
whole pipeline.
The architecture docs, extension and library API guides, and the docs
that link source files name the new crates and paths. Links pinned to
a commit keep their old paths.
@zzylol zzylol added documentation Improvements or additions to documentation ir Pre-ASAP or post-ASAP intermediate representation optimizer Workload optimization and plan selection labels Oct 18, 2026
@zzylol
zzylol merged commit 58999df into main Oct 18, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation ir Pre-ASAP or post-ASAP intermediate representation optimizer Workload optimization and plan selection

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reorganize crates and modules by #509 stages and the #511 unified IR

1 participant