Skip to content

[Core] Add predicate deletes to TableUpdate - #1105

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-predicate-delete
Oct 12, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-predicate-delete

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

PyPaimon already supports delete_by_predicate, but native updates expose only row-ID deletion. Python still reads partition manifests or materializes matched row IDs before invoking Rust. Move that complete existing operation into Rust core.

Changes

  • Add TableUpdate::delete_by_predicate, with batch and stream Python wrappers and updated type stubs.
  • For partition-only predicates, prepare deletion messages for all active data and index files in the selected partitions. This supports append, primary-key and data-evolution tables, including pending bucket -2 files, without requiring deletion vectors.
  • For other predicates, stream matched row IDs into the existing data-evolution deletion-vector writer. Match reading and DV preparation share one captured snapshot; FULL scalar-index mode includes unindexed appends.
  • Extract the committer's existing complete snapshot-manifest reader for shared internal use. No new public scan abstraction or dependency is introduced.
  • Validate authorization, time travel and predicate field/schema consistency before reading metadata. Publication remains the caller's responsibility.

Java semantics

Reuse Java-compatible commit-message baselines, anchor selection and DV replacement checks. Default file/DV/row-ID conflict detection preserves later appends with fresh row IDs and rejects conflicting replacements. Java's stricter partition-level checker is configured separately through an explicit last-safe snapshot; a message baseline alone does not enable it. The paired Python change removes its extra coarse partition check.

Verification

  • Reproduced the missing native method with a binding test before implementation.
  • 53 focused core update/delete tests passed, including 11 new predicate-delete tests.
  • 121 core commit tests passed.
  • 21 binding update/delete tests passed.
  • Paired PyPaimon integration: 756 related Native tests plus 2 additional REST snapshot tests passed; final predicate suite 30 passed.
  • cargo +1.98.0 clippy -p paimon -p pypaimon_rust --all-targets -- -D warnings and cargo fmt --all --check passed.
  • Independent core and wrapper/Java-semantics reviews found no blocking issues.

Paired PyPaimon adapter and Java-semantics alignment: apache/paimon#10507.

@QuakeWang QuakeWang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the Rust core and Python binding changes. Predicate deletes capture one snapshot, correctly separate partition metadata deletion from row-level deletion-vector preparation, include pending/index files, and preserve conflict baselines. The focused tests and all Rust CI checks pass. No blocking issues found.

@JingsongLi
JingsongLi merged commit f282f58 into apache:main Oct 12, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants