Repository navigation
[core] Add confirmed snapshot commit callbacks - #1108
Merged
JingsongLi merged 2 commits intoOct 12, 2026
Merged
Conversation
leaves12138
approved these changes
Oct 12, 2026
leaves12138
left a comment
There was a problem hiding this comment.
LGTM. I reviewed the confirmed-publication callback flow, lost-response recovery from persisted delta manifests, row-tracking metadata assignment, overwrite/delete entries, callback error propagation, and Python callback resource ownership. The targeted Rust suite passed (125 tests), and all Rust PR checks are green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
PyPaimon already supports commit callbacks, but registering one currently forces its REST commit path back to Python because Rust core cannot deliver a post-publication notification.
Changes
CommitCallbackandCommitCallbackContextto Rust core. Context contains the confirmed snapshot, its data-file delta and the commit identifier, including assigned row IDs and sequence numbers.add_commit_callback(callback)on the Python batch and stream committers. Callbacks receive a typed, retainableCommitCallbackContextwrapping core snapshot, manifest entries, file metadata and statistics. Preserve original Python exceptions and allow Native reads inside callbacks.The notification timing and data-file delta follow Java
FileStoreCommitImpl/CommitCallback.Context. The exposed context covers existing PyPaimon capabilities. Java's separate checkpointretry(committable)hook is outside this change; filtering already committed identifiers does not emit a notification. Format Tables have no Paimon snapshots and reject registration.Python binding API
The context and metadata views are read-only and can be retained after the callback returns. Callback ownership remains with the caller; the native committer releases its Python reference on close.
Verification
cargo test -p paimon --lib table::table_commit::tests --locked: 125 passed.test_table_commit.py: 71 passed.fulltext,vortexand-D warnings: passed.The dependent PyPaimon PR apache/paimon#10509 retains the existing public callback API and continue building Rust
mainin CI.