Skip to content

[Python] Commit snapshot properties through Native - #10505

Merged
JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/native-snapshot-properties
Oct 11, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/native-snapshot-properties

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Summary

PyPaimon already supports caller snapshot_properties in batch, stream and overwrite commits. The presence of an explicit property map previously forced the Python publication path, even for {} and REST tables which otherwise qualified for Native.

  • Route eligible commits with properties through Native and pass metadata to the existing batch/stream commit operation.
  • Keep callback and non-REST routing on the Python path.
  • Preserve the preparation/publication exception boundary: errors once Native publication begins must not trigger another publication or abort prepared files.
  • Add REST tests for Append, PK and Data Evolution, stream checkpoint isolation, empty commits, static/dynamic empty overwrite and lost publication responses.

Properties describe the current commit, are not inherited by subsequent checkpoints, and empty maps are absent from snapshot metadata. These rules follow Java ManifestCommittable.properties and FileStoreCommitImpl snapshot construction. No new Python capability or API is introduced.

Dependency

Depends on apache/paimon-rust#1104 for Rust core snapshot-property publication and the optional binding argument. This PR is draft until that change reaches Rust main. Native CI continues to build upstream apache/paimon-rust@main.

Verification

Built the companion Rust extension and enabled Native plan/read/write/commit for:

pypaimon/tests/native_snapshot_properties_test.py
pypaimon/tests/native_commit_test.py
pypaimon/tests/table_commit_test.py
  • 117 tests passed, with 68 Native plans, 56 reads, 54 writes and 73 commits exercised.
  • Existing Python table_commit_test.py and table/simple_table_test.py: 41 passed.
  • Rust commit regression: 121 passed; Python binding commit suite: 58 passed.
  • Clippy 1.98.0, Rust formatting, Python Flake8, Python 3.6 syntax parsing and diff checks passed.
  • Independent core/API and Python adapter reviews have no remaining findings.

New cases assert that property maps cannot leak between stream checkpoints, empty-map metadata is omitted, static empty overwrite publishes properties while dynamic empty overwrite creates no snapshot, and publication response loss preserves both metadata and referenced data files.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at e43cb4f. The Native routing preserves the existing preparation/publication exception boundary, passes snapshot properties per call, and keeps callback/non-REST fallback behavior. The tests cover append, PK, Data Evolution, stream checkpoint isolation, empty commits, static/dynamic overwrite, and publication response loss. No blocking correctness issue found. This remains dependent on Rust #1104 reaching main, and the latest Native CI was still running at review time.

@JingsongLi
JingsongLi marked this pull request as ready for review October 11, 2026 11:49
@JingsongLi
JingsongLi merged commit cd95f04 into apache:master Oct 11, 2026
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants