[Python] Delegate local vindex builds to native Rust

[JingsongLi]

Purpose

Delegate PyPaimon's existing local REST vindex builds to the common Rust global-index builder for ivf-flat, ivf-pq, ivf-sq, ivf-rq and diskann.

Depends on apache/paimon-rust#1093 for NULL vectors, sparse relative row IDs, all-NULL shards and Java-compatible training defaults. This PR remains draft until that Rust change is merged. CI continues to install Rust from main.

Changes

• Enable the existing native route for the five vindex families on eligible REST/Parquet data-evolution tables.
• Reuse Python's existing Java-compatible option mapping and training-ratio parsing before forwarding options. Rust owns row scanning, snapshot/range planning, nullable-vector compaction, training and index file generation.
• Keep prepare/commit ownership: build returns unpublished messages, native failures propagate, and submitted/returned files remain available after uncertain commits.
• Replace the dense-vector fallback test with end-to-end sparse-vector native tests.

Verification

PYPAIMON_TEST_NATIVE_PLAN=1 .venv/bin/python -m pytest pypaimon/tests/native_vector_index_build_test.py pypaimon/tests/native_generic_index_build_test.py pypaimon/tests/native_sorted_index_build_test.py pypaimon/tests/global_index_build_test.py -q --tb=short

205 passed and 19 subtests passed, with 55 native plans exercised, against the paired Rust implementation.

Coverage includes:

• Both Python and Rust readers return original sparse row IDs for all five families.
• All-NULL shards create no index file or snapshot; later shards and incremental builds retain correct global IDs.
• Sparse full-spill files match the existing Python writer byte for byte, including training ratios, automatic sizing, PQ defaults, field aliases, OPQ and tuning budgets.
• Real valid-vector cardinality drives automatic PQ size-budget validation; all-NULL input skips native training.
• Partition filtering, external index placement and both committers continue to work; source conflicts preserve returned files.
• Invalid non-NULL vectors and invalid build parallelism fail without retrying through the Python row writer.

Flake8, Python 3.6 syntax checks and git diff --check passed.

Scope

The public Python build/commit APIs remain unchanged. Local native builds use the existing eligibility checks; Ray execution retains its current implementation.

[leaves12138]

Reviewed 971cda9c against a locally rebuilt apache/paimon-rust#1093 extension at 94a8eb0d. No blocking findings.

Reusing native_options/train_sample_ratio preserves Java-compatible mapping, field/build precedence and fixed VECTOR dimensions rather than exposing Rust-only raw aliases through the public Python API. Native construction retains unpublished messages, sparse source row IDs, logical row coverage and caller commit ownership; real build errors propagate without a second Python build. Local eligibility and the existing Ray path remain separate.

Independent local validation with Python 3.10 and PyArrow 19.0.1:

• Paired REST/index-build suites: 205 passed and 19 subtests passed, with 55 Native plans exercised.
• Seven related vector/index/search suites: 225 passed and 59 subtests passed, with 49 Native plans exercised.
• Additional reviewer tests: 30 passed across all five vindex families for fixed-size nullable VECTOR shards, dimension precedence, updates to NULL with both committers, conflict ownership and multi-shard private-output cleanup.
• Rebuilt binding regression: 417 passed with a Native-generated smoke warehouse.
• Flake8 using dev/cfg.ini and Python 3.6 syntax parsing passed.

The paired tests verify sparse full-spill bytes against the existing Python writer, automatic sizing/budgets, all-null and incremental shards, and original sparse IDs through both Python and Rust readers.

Scope: Java consistency was source/byte-contract comparison, not a Java engine or production-storage interoperability run. Native shard preparation is currently sequential despite validation of global-index.build.parallelism. Keeping this PR draft until Rust #1093 reaches main is appropriate; Native CI against an older main does not validate this new sparse-build dependency.