Skip to content

feat(index): support partition-scoped global index drops - #1106

Open
QuakeWang wants to merge 2 commits into
apache:mainfrom
QuakeWang:index-partition-drop
Open

QuakeWang wants to merge 2 commits into
apache:mainfrom
QuakeWang:index-partition-drop

Conversation

@QuakeWang

Copy link
Copy Markdown
Member

Purpose

drop_global_index declared partitions but rejected it, while the core builder could only drop matching indexes across all partitions.

Brief change log

  • Add core partition predicates and structured SQL partition parameters.
  • Reject invalid parameters before deletion; preserve dry-run and snapshot-guarded commits.
  • Reuse Java FLOAT formatting and preserve signed-zero and NaN partition matching.

Tests

80 focused tests passed, covering partition selection, retained index definitions, invalid parameters, floating-point regressions, dry-run, and stale snapshots. CI-pinned formatting, strict Clippy, dependency and license checks passed.

API and Format

Adds GlobalIndexDropBuilder::with_partition_filter. No storage format changes.

Documentation

Document structured partitions syntax in the procedure module.

Add a partition predicate to the core drop builder and parse structured
partition specifications in the SQL procedure. Reject invalid parameters
before deletion and preserve dry-run and snapshot-guarded commits.

Reuse Java-compatible FLOAT formatting and compare floating partition keys
with signed-zero and NaN semantics. Cover partition selection, unchanged
index definitions, invalid inputs, and non-committing previews.

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED. Implementation: FINDINGS at 47b42b4. Partition-scoped index maintenance has end-to-end value, but the new SQL parser can drop a different LTZ partition's index from the one requested.

Validated against the integration merge with main ae858ea: all 49 original SQL procedure tests, 17 core drop tests, 39 partition-format tests, and the stale-snapshot commit guard pass. An independent actual SQL write/build/select/selective-drop probe also preserves query results. A second actual SQL probe under TZ=Asia/Shanghai, with nonlegacy LTZ partitions, reproduces the wrong-partition deletion described inline. Please cover this supported timezone/partition-format combination before production use.

Comment thread crates/integrations/datafusion/src/procedures.rs Outdated
Parse non-legacy LTZ partition values in the system-local timezone while
preserving UTC semantics for legacy names and explicit timezone offsets.
Accept legacy names with omitted zero seconds and reject ambiguous or
nonexistent local times before deletion.

Cover SQL deletion in isolated timezones, timestamp precision, retained
index definitions, invalid parameters, and non-committing dry runs.

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
@QuakeWang
QuakeWang requested a review from JingsongLi October 12, 2026 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants