Repository navigation
refactor!: arrange crates by planning stage - #741
Merged
Merged
Conversation
asap-types is arranged by IR section: ir::{operator, scalar, schema,
properties}, workload and physical replace pre_asap, post_asap, types,
resources and parsed_workload. Materialization and Medium move to
physical::materialization; ExecutionTiming and the data-state types to
ir::properties::execution_data_state. The unused query-time error
estimation moves to the executor.
asap-aware-mapping is split into one crate per stage:
asap-logical-asap-optimization (Stage 1: Pass 1, Pass 2, whole-workload
candidate enumeration, analytical accuracy), asap-physical-asap-optimization
(Stage 2) and asap-plan-selection (Stage 3, its cost modules, PlanningModels
and the Stage 1 to Stage 3 driver). asap-physical-operators is renamed
asap-executor. Cargo now enforces the one-way stage flow, so the
source-scanning cost_independence test is removed.
Paths, use lines and manifests only. Items a later stage reads become pub,
the shared test helpers sit behind Stage 1's test-support feature, and
cms_summary reads Stage 1 candidates directly instead of running the
whole pipeline.
The architecture docs, extension and library API guides, and the docs that link source files name the new crates and paths. Links pinned to a commit keep their old paths.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The code layout still followed the pre-#509 split.
asap-typeshadpre_asap/post_asapmodules whose names no longer described their contents, andasap-aware-mappingheld Stages 1, 2 and 3 in one crate, so nothing stopped Stage 1 from importing Stage 3 code. #572 (approved 2026-10-03, with the crate names decided 2026-10-13) arranges the crates by planning stage, so that Cargo enforces the one-way stage flow, and arrangesasap-typesby #511 IR section.What
This is a pure move: file paths,
uselines,Cargo.tomlfiles and the docs that name crate paths change. No planning logic changes, and selections and costs are unchanged.asap-typeskeeps its name.pre_asap,post_asap,types,resourcesandparsed_workloadare gone. Their contents now live underir::{operator, scalar, schema, properties},workloadandphysical.asap-aware-mappingis split into three stage crates:asap-logical-asap-optimization(Stage 1),asap-physical-asap-optimization(Stage 2) andasap-plan-selection(Stage 3).asap-physical-operatorsis renamedasap-executor(Stage 4). No stage crate depends on it.asap-plannerfacade re-exports fromasap-plan-selection, as it did before fromasap-aware-mapping::pass.How
The moves were done with a script:
git mvevery file.usetree and inline path (crate::,super::, extern) against the file's old module and maps it to its new module. It keeps the text as written when the path still resolves, and rewrites the path otherwise.lib.rs, the newmod.rsfiles) were written by hand.pub(list below).Cycles and how each was resolved:
pass/) runs all three stages, so it goes into plan-selection withPlanningModels, not into the facade. Stage 3's unit tests callrun_stage1_to_stage3, and they could not do that from a crate below the facade. The facade re-exports it.stage1_logical_asap_optimization,Stage1Candidate,Pass2Candidate, and the private Pass 2 driverPass2/Rules/Pass2List/pass2_candidates/next_choice) moved out ofpass/stage1_to_stage3.rsinto Stage 1's newworkload_candidates.rs, together with the test that covers them. It called Stage 2'sfixed_interval_ms, a pure function ofQueryRecurrence, so that function moved with it, and Stage 2 now imports it from Stage 1.test_supportis used by the tests of all three stage crates. It now lives in Stage 1 aspub mod test_supportbehind thetest-supportfeature. The later stage crates enable that feature in their dev-dependencies.cms_summaryused to run the whole pipeline only to read its Stage 1 candidates. It now callsstage1_logical_asap_optimizationon the same pruned Pass 1 inventory, with the same built-in deployment and recurrence, so it returns the same node.ExecutionTimingandMaterializationwere in one file.MediumandMaterialization(the Stage 2 decision) moved totypes::physical::materialization.ExecutionTiming,ExecutionDataState,DataPrimitiveandExecutionDataStateErrormoved totypes::ir::properties::execution_data_state, as Reorganize crates and modules by #509 stages and the #511 unified IR #572 recorded.Old path → new crate / module
asap_types::ir::{node, non_asap, asap, operator_properties, query, pane_chain}asap_types::ir::operator::*(their::re-exportsOperatorNode,ASAPOp,QueryRoot, … stay)asap_types::pre_asap::agg_intent,post_asap::maintained_populationasap_types::ir::operator::{agg_intent, maintained_population}asap_types::pre_asap::{expr_ir, scalar_type_rules, column_resolution}asap_types::ir::scalar::*asap_types::pre_asap::schemaasap_types::ir::schemaasap_types::post_asap::{sketch, family_capability},ir::{aggregate_schema, error, schema_support}asap_types::ir::schema::*asap_types::types(AccuracyTarget,AccuracyMetric, …)asap_types::ir::properties::accuracy(re-exported fromir::properties)asap_types::post_asap::guarantee,ir::{timing, summary_coverage}asap_types::ir::properties::*asap_types::post_asap::execution_data_stateir::properties::execution_data_stateandphysical::materialization(Materialization,Medium)asap_types::post_asap::summary_windowasap_types::physical::summary_windowasap_types::post_asap::query_time(posterior error-bound math, no callers)asap_executor::query_timeasap_types::{workload, parsed_workload, resources}asap_types::workload,workload::parsed_workload,workload::resourcespre_asap::X/post_asap::Xre-exportsX:ir::operator::X,ir::scalar::X,ir::schema::X,ir::properties::X,physical::Xasap_aware_mapping::{per_computation_candidates, asap_aware_cse, realization, accuracy, deployment_capabilities, maintained_population, function_rules, test_support}asap_logical_asap_optimization::*asap_aware_mapping::pass::{stage1_logical_asap_optimization, Stage1Candidate, Pass2Candidate}asap_logical_asap_optimization::workload_candidates::*asap_aware_mapping::stage2_physical_asap_optimizationasap_physical_asap_optimization(crate root)asap_aware_mapping::stage3_plan_selection(+measured_operation_costs)asap_plan_selection(crate root)asap_aware_mapping::{analytical_cost, physical_operator_statistics, physical_handoff_cost, query_physical_lowering, storage_io}asap_plan_selection::cost::*asap_aware_mapping::pane_sharingasap_plan_selection::pane_sharingasap_aware_mapping::pass(optimize,OptimizationPass,PassRegistry,PlanningModels,run_stage1_to_stage3, …) and its root re-exportsasap_plan_selection::passandasap_plan_selection::*asap_aware_mapping::{AccuracyModel, DeploymentCapabilities, summary_candidates, …}root re-exportsasap_logical_asap_optimization::*crates/asap-aware-mapping/tests/per_computation_candidates.rscrates/logical-asap-optimization/tests/crates/asap-aware-mapping/tests/{physical_handoff_cost, storage_io}.rscrates/plan-selection/tests/crates/asap-physical-operators(asap_physical_operators)crates/executor(asap_executor)Crate dependency graph (normal dependencies)
Dev-dependencies only: the frontends' tests and the executor's tests use Stage 1 and plan-selection. Stages 2 and 3 enable
asap-logical-asap-optimization/test-support. The facade's tests use Stage 1.Visibility changes (
pub(crate)→pub), needed by a later stage crateaccuracy::composition:INPUT_ROW_COUNT,DIVISION_OPERAND_DOMAINS,PRODUCT_OPERAND_DOMAINS,STREAM_L1_NORM,samples_sign_statistic,parse_samples_sign_statistic,parse_source_norm_statisticaccuracy::estimators:with_failure_probability;hll::{classic_relative_error, classic_failure_statistic, parse_classic_failure_statistic, classic_failure_probability, min_precision};univmon::{parse_incomplete_top_layer_statistic, incomplete_top_layer_probability}per_computation_candidates:approximate_inputs,root_guarantee,with_resized_summaries,distinct_bound_source,project_column_guarantee,operand_of,operand_guarantee,operand_domain_statisticrealization::ranked_columnworkload_candidates::fixed_interval_ms(waspub(crate)in Stage 2)test_supporthelper (feature-gated, test-only)Key code interfaces added / modified
Module paths only. No type, field, function signature or wire format changes. The module paths are listed in the table above. Crates added, renamed or removed:
Usage: a downstream crate that planned through
asap_aware_mappingnow depends onasap-plan-selection. For example,tools/verify_metricsql_external_consumer.shlowers MetricsQL withasap_types::ir::properties::AccuracyTarget::exact(), which wasasap_types::types::AccuracyTarget::exact().asap-planner's API is unchanged.Deviations from #572
logical-asap-optimization,physical-asap-optimization,plan-selection), not Reorganize crates and modules by #509 stages and the #511 unified IR #572'slogical-optimizer/physical-optimizer. Inside Stage 1, Pass 1 and Pass 2 keep their module names,per_computation_candidatesandasap_aware_cse, rather thanpass1/andpass2/.pass/driver andPlanningModelsare in plan-selection, not in the facade, for the reasons under How. The facade re-exports them.AccuracyModelstays in Stage 1. Pass 1 still callsDefaultAccuracyModelthrough theAccuracyModeltrait (eight calls inper_computation_candidates.rs). Moving the trait to plan-selection needs those calls changed to the inherent estimator functions, which is a code change, not a move. The trait andAccuracyEvidenceProviderstay inasap_logical_asap_optimization::accuracy, and plan selection uses them throughPlanningModelsas before. That change is left for a follow-up PR.plan-selection/accuracy/.erpandempirical_comparisonwere already deleted before this PR.pane_sharing(to plan-selection) andquery_time(to the executor) are moved, not deleted. Deleting them belongs to the PRs that replace them.pre_asap_sub_dagexport key,tools/dag-viewer/post_asap_fixture.json, and the integration-test helper moduleasap_integration_tests::post_asap.asap-aware-mapping/tests/cost_independence.rsscanned source files to check that Stage 1 and Stage 2 import no cost module. Cargo now enforces this: neither crate depends onasap-plan-selection. Its 2 tests are the only ones removed (1803 → 1801#[test]s).Part of #572. It does not close #572: the
AccuracyModelmove and the leftover names above remain.Before this PR
All of Stages 1–3 lived in
asap-aware-mapping, and stage boundaries were held only by a source-scanning test.asap-typesexposedpre_asap,post_asapandtypesmodules that cut across #511's sections.After this PR
Each stage is its own crate. A Stage 1 or Stage 2 change that imports Stage 3, or any stage that imports the executor, fails to build.
asap-typesis arranged asir::{operator, scalar, schema, properties},workloadandphysical. The architecture doc, the extension guide and the library API guide name the new crates and paths.Verification
Run locally at the PR head:
cargo check --workspace --all-targets: clean, no warnings.cargo fmt --all -- --check: clean.cargo clippy -p asap-types -p asap-logical-asap-optimization -p asap-physical-asap-optimization -p asap-plan-selection -p asap-executor -p asap-planner --all-targets --all-features --locked -- -D warnings: clean.cargo nextest run -p asap-types -p asap-logical-asap-optimization -p asap-physical-asap-optimization -p asap-plan-selection -p asap-executor -p asap-planner -p asap-devtools: 901 passed, 7 skipped.cargo nextest run -p asap-integration-tests -E 'binary(/planner_layering/) | binary(reuse_rules) | binary(stage1_logical_asap_optimization) | binary(materialization_e2e) | binary(stage2_physical_asap_optimization_sql) | binary(deployment_capabilities) | binary(promql_to_post_asap)': 129 passed, 21 skipped (#[ignore]tests).cargo test --docfor the moved crates (the executor README doctest): passed.#[test]s, which are the 2 deletedcost_independencetests.tools/verify_metricsql_external_consumer.shwas updated to the new path but not run locally. CI runs it.