Skip to content

[python] Delegate predicate deletion to native TableUpdate - #10507

Merged
JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/native-predicate-delete
Oct 12, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/native-predicate-delete

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Purpose

Use native execution for PyPaimon's existing BatchTableUpdate and StreamTableUpdate.delete_by_predicate. Previously Python still planned partition metadata deletion or collected all matched row IDs before invoking the native row-ID writer.

Depends on apache/paimon-rust#1105. Keep this draft until that PR reaches Rust main; Native CI continues to build upstream main.

Changes

  • Delegate the complete predicate-delete operation to Rust core, converting only predicates and commit messages in Python.
  • Keep native preparation separate from execution: execution, snapshot-load and message-conversion failures cannot fall back into a second Python operation.
  • Share common native update eligibility. Partition-only deletes also use Rust for append/PK tables without row tracking or deletion vectors, including pending bucket -2 files.
  • Align Python commit conflict checks with Java: remove the extra partition-level rejection based solely on a message snapshot baseline. Prepared deletes preserve subsequent appends with fresh row IDs; existing file/DV/row-ID checks still reject conflicting replacements and updates.
  • Add REST tests for batch/stream operation ownership, partition/data/mixed/NULL predicates, column deltas, prior DV contents, scalar indexes, unindexed rows, pending files, races and snapshot-load errors.

Java semantics

FileStoreCommitImpl.configureRowIdCheckFromMessages configures row-ID checking; it does not enable StrictModeChecker. The stricter checker is constructed only when an explicit last-safe snapshot is configured. DataEvolutionConflictDetection ignores fresh row IDs beyond the baseline and checks overlapping historical row/column writes. Python now follows this default behavior.

Verification

Built the paired Rust extension locally and enabled Native planning, reads, writes and commits:

  • 756 related Native regression tests passed; 2 additional REST snapshot tests passed.
  • Final predicate-delete suite: 30 passed, including after rebasing onto current master.
  • Existing Python update suite: 130 passed plus 20 subtests.
  • Existing commit/conflict suites: 93 passed plus 13 subtests.
  • Changed-file flake8 and Python 3.6 syntax checks passed.
  • Independent core and facade/Java-semantics review found no blocking issues; extra batch/stream checks confirmed concurrent existing-row updates still conflict.

Prepared files remain preserved on commit failures or unknown outcomes.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the Python facade and Java-semantics alignment. Native predicate deletes keep preparation failures on the fallback path while operation failures propagate, support batch/stream and partition/row predicates, and preserve snapshot/conflict semantics. The focused Python tests and all completed CI checks pass; Native CI is still pending. No blocking issues found.

@JingsongLi
JingsongLi marked this pull request as ready for review October 12, 2026 00:21
@JingsongLi
JingsongLi merged commit 02d2cca into apache:master Oct 12, 2026
14 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants