Skip to content

[core] Add confirmed snapshot commit callbacks - #1108

Merged
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-commit-callbacks
Oct 12, 2026
Merged

JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-commit-callbacks

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Why

PyPaimon already supports commit callbacks, but registering one currently forces its REST commit path back to Python because Rust core cannot deliver a post-publication notification.

Changes

  • Add CommitCallback and CommitCallbackContext to Rust core. Context contains the confirmed snapshot, its data-file delta and the commit identifier, including assigned row IDs and sequence numbers.
  • Notify in registration order only after confirmed publication. Recover notifications from persisted delta manifests after a lost publication response, retaining file statistics.
  • Propagate callback errors while preserving published data and metadata. Callback failures stop later callbacks and do not republish the snapshot.
  • Expose public add_commit_callback(callback) on the Python batch and stream committers. Callbacks receive a typed, retainable CommitCallbackContext wrapping core snapshot, manifest entries, file metadata and statistics. Preserve original Python exceptions and allow Native reads inside callbacks.
  • Keep the existing Java v14 bridge and Snapshot JSON conversion internal to the PyPaimon adapter; neither is part of the public callback signature.
  • Make callback references visible to Python GC and release them on close. Cover callback-owner cycles with both explicit close and abandoned committers.

The notification timing and data-file delta follow Java FileStoreCommitImpl / CommitCallback.Context. The exposed context covers existing PyPaimon capabilities. Java's separate checkpoint retry(committable) hook is outside this change; filtering already committed identifiers does not emit a notification. Format Tables have no Paimon snapshots and reject registration.

Python binding API

commit.add_commit_callback(lambda context: print(
    context.identifier,
    context.snapshot.id(),
    [entry.file.file_name for entry in context.commit_entries],
))

The context and metadata views are read-only and can be retained after the callback returns. Callback ownership remains with the caller; the native committer releases its Python reference on close.

Verification

  • cargo test -p paimon --lib table::table_commit::tests --locked: 125 passed.
  • Python binding test_table_commit.py: 71 passed.
  • PyPaimon Native write / commit / callback integration: 187 passed against this branch's locally built extension.
  • Rust 1.98.0 formatting and workspace Clippy with fulltext,vortex and -D warnings: passed.
  • Independent reviews of core publication/recovery and Python bridge/resource ownership completed; the GC finding was fixed and revalidated.

The dependent PyPaimon PR apache/paimon#10509 retains the existing public callback API and continue building Rust main in CI.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I reviewed the confirmed-publication callback flow, lost-response recovery from persisted delta manifests, row-tracking metadata assignment, overwrite/delete entries, callback error propagation, and Python callback resource ownership. The targeted Rust suite passed (125 tests), and all Rust PR checks are green.

@JingsongLi
JingsongLi merged commit 9745230 into apache:main Oct 12, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants