Skip to content

[Python] Keep REST commits with callbacks on the native path - #10509

Merged
JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-commit-callbacks
Oct 12, 2026
Merged

JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-commit-callbacks

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Why

PyPaimon supports commit callbacks, but registering a callback forces REST commits to use the Python implementation even when Native commit is enabled.

Changes

  • Register the existing callbacks through Native's public add_commit_callback interface and keep eligible REST commits on the Native publication path.
  • Adapt the typed Native CommitCallbackContext to the existing Python CommitCallbackContext, including Python Snapshot, ManifestEntry, GenericRow, canonical file paths and assigned row-tracking metadata.
  • Decode one context per notification and share it across callbacks in registration order. Callbacks added after Native initialization or between commits take effect on later commits.
  • Preserve the existing callback close lifecycle, propagate callback exceptions, and retain committed files after notification failures or lost publication responses.

Publication and recovery notification belong to Rust core; Python adapts the existing API. The public Python callback API is unchanged.

Verification

  • Native commit / write / existing callback regression suites: 187 passed.
  • Native counters: 127 plans, 115 reads, 86 writes and 88 commits exercised.
  • Coverage includes append, primary-key and Data Evolution tables; overwrite ADD/DELETE entries; empty commits; lost REST responses; callback errors; Native reads inside callbacks; late registration; shared contexts; close errors and callback-owner GC cycles.
  • Flake8 and Python 3.6 grammar validation for changed Python sources: passed.

Dependency

Requires apache/paimon-rust#1108. This PR is draft until the Rust change merges; Native CI continues building upstream Rust main.

@leaves12138

Copy link
Copy Markdown
Contributor

LGTM from the code review. The native callback bridge preserves the existing Python context, callback ordering/shared context, late registration, recovery, error propagation, and close/GC lifecycle. The Python Native CI failures are expected until dependency PR apache/paimon-rust#1108 is merged: that job currently builds Rust main, so _add_commit_callback is unavailable and the new tests intentionally fail on Python fallback. Please rerun Native CI after #1108 lands.

@JingsongLi
JingsongLi marked this pull request as ready for review October 12, 2026 08:54
@JingsongLi
JingsongLi merged commit e36ed84 into apache:master Oct 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants