[python] Use native builders for local sorted global indexes

[JingsongLi]

Purpose

The core API in apache/paimon-rust#1088 has merged. Native CI also requires apache/paimon-rust#1090 for fixed-size vector normalization, early unsupported-format detection and correct vector ranges across multiple file groups. Draft until that CI fix is available on Rust main.

Use Rust for PyPaimon's existing local BTree and Bitmap global-index builds on eligible REST Data Evolution tables. Snapshot selection, range planning, reading, sorting and file generation stay in Rust core; Python retains the existing GlobalIndexBuilder and CommitMessage API.

Changes

• Dispatch local sorted-index builds when write.native.enabled is enabled and the table/binding is eligible.
• Pass the existing partition filter and user options through the native builder and reuse the commit-message serializer bridge.
• Keep authorization checks on the existing Python build path. Native build errors propagate once a build starts.
• Read Java-compatible DataEvolutionIndexSourceMeta and validate staged index sources with the Python committer too: reject target-column changes, deleted column deltas, missing/future snapshots and removed row ranges. Pure appends and unrelated column updates remain valid.
• Preserve prepared files after commit exceptions, including a successfully published snapshot whose response is lost. Explicit TableCommit.abort remains the caller's choice.
• Add REST integration coverage for BTree/Bitmap, partitions and incremental builds, external paths, typed keys, both committers, NULL/empty-set partition predicates, authorization and failure paths.

Validation

• PYPAIMON_TEST_NATIVE_PLAN=1 pytest native_sorted_index_build_test.py native_vector_search_test.py global_index_build_test.py write/conflict_detection_test.py file_store_commit_test.py: 283 passed, 32 subtests; 104 native plans exercised
• Final native_sorted_index_build_test.py: 87 passed
• Flake8 with dev/cfg.ini and Python 3.6 syntax checks passed

CI continues to install paimon-rust from main. Merge the Rust counterpart first so these native tests exercise the new API on CI.

Native CI follow-up

• The failing run installed Rust commit 637386b196da0b017557a1fdc01c518834732fb9. All 149 failures and 195 setup errors initially failed at FixedSizeList input normalization.
• Require a supported physical file suffix for Native reads; a .vector. name alone does not make Lance or Vortex available in the default Rust wheel. Unsupported formats retain the Python reader path before any Native read begins.
• Add REST vector write regression coverage for Batch/Stream, Append/Data Evolution, FLOAT/DOUBLE, child aliases, slices, NULL vectors and NULL elements; verify unavailable vector formats fall back before accepting data.
• Supply bounded Arrow batches in dedicated row-rolling tests; restore every required Blob column in ACT full-row append fixtures; compare unordered vector lookup rows without assuming physical file order.
• Exercise Native nested ROW projection separately from the Python reader's unsupported projection contract.

Verification with the rebuilt Rust fix

• All five CI Native flags: 1090 passed, 13 subtests passed. Includes the previously failing CI cases and related writer, reader, update, merge and sorted-index suites.
• Native operations exercised: plans=2317, reads=2119, writes=1124, commits=76; updates: row_id=88, grouped=50, predicate=47, upsert=11, incremental=7, merge=24.
• Focused reader/writer/ACT/batch-vector/rolling suites: 251 passed, 2 subtests passed.
• Flake8 and Python 3.6 syntax checks passed for all six changed follow-up files.

CI still installs Rust main; merge apache/paimon-rust#1090 before rerunning Native CI.

[leaves12138]

Reviewed this head against the rebuilt apache/paimon-rust#1088 extension at c850b07f9fc3537b720b204aa61d2c0255d6582e. No new blocking issue was found in this PyPaimon change.

The existing build-then-commit API is preserved. The Native dispatcher retains the capability and authorization gates, propagates errors after a Native build starts, and does not launch a second Classic build as a fallback. Both committers validate the prepared source on every attempt and preserve prepared files on commit exceptions, including uncertain publication outcomes. The metadata encoding, sorted-index options and nullable set-predicate behavior were checked against Java source.

Verification with both PR heads:

• Affected PyPaimon REST/index/commit suites: 283 passed plus 32 subtests.
• Related Native suites: 997 passed plus 96 subtests.
• Reviewer-added cases: 16 passed: eight publication-conflict retries across BTree/Bitmap and both committers, plus eight Native empty/nullable IN/NOT IN read cases. Target-column updates injected during the first publication attempt are rejected on retry; unrelated-column updates can retry and publish.
• Configured Flake8 for the six changed Python files and Python 3.6 syntax checks passed.

Merge-order condition: keep this Draft until the corrected Rust counterpart is on main, since Python Native CI continues building Rust main. I requested changes on #1088 only for its stale binding test that still expects empty IN to raise; the new acceptance behavior is correct and should not be reverted.

Java was used as a source-level semantic reference, not run in a separate process. No production-service integration was performed.

[leaves12138]

Follow-up CI status for this unchanged head, checked alongside Rust #1088 at afbbb1684823cac65f95d121c3c003a80b517e68: my existing code-review approval stands, but this does not mean the Native CI is green.

I downloaded the failed Native CI job log. It built Rust main at 637386b196da0b017557a1fdc01c518834732fb9, with PyArrow 19.0.1. The failures include FixedSizeList child-field schema mismatches while writing vector data (the expected child has the name element, while the actual child has no name). These are not simply missing-new-builder-API errors, and I have not established that this PR introduces them. The new sorted-builder test module is separately gated on the builder method being available.

With the reviewed counterpart extension, the affected REST/index/commit suites pass locally (283 tests plus 32 subtests, 104 Native plans exercised), and the independent commit-retry/set-predicate matrix passes (16 tests). The Rust follow-up only updates tests and is now approved.

Please rerun Native CI with the corrected Rust counterpart available on main, and ensure the broader Arrow/schema failures are resolved before merging this dependent Python PR. Do not treat the current failed job as verified coverage of the new sorted-builder path.