Support sparse vindex builds with Java-compatible training semantics

[JingsongLi]

Purpose

Complete native vindex construction for behavior already supported by PyPaimon and Java, so PyPaimon can delegate local REST vector-index builds to Rust.

Java NativeVectorGlobalIndexWriter skips NULL vectors, preserves their logical row coverage and adds the original shard-relative IDs for non-NULL vectors. The previous Rust pipeline required dense vectors, so the Python adapter excluded all five vindex families.

Changes

• Compact nullable Arrow List / FixedSizeList batches while validating every logical row ID. Retain Arrow storage for dense batches; ignore child payloads hidden behind NULL parents and reject invalid non-NULL vectors.
• Spill original relative IDs separately from compacted float vectors. Train on evenly spaced samples of the non-NULL stream, with bounded read buffers; retain full logical row_count and row ranges in manifest metadata.
• Produce no file or commit for all-NULL shards. Granule builds that encounter NULL vectors drain private worker tasks before preparing through full spill against the pinned snapshot.
• Align automatic nlist, inferred PQ width/OPQ, option aliases, field/build precedence and tuning objectives with Java. Compile the native configuration when the real valid-vector count is known; avoid synthetic-count budget failures.
• Preserve documented raw native aliases and strict foreign-family option rejection for explicit Rust/SQL builder options. Keep PK query dimension aliases working.
• Accept and validate Python's build parallelism option; native shard preparation currently remains sequential.
• Update SQL configuration and NULL-vector documentation.

Reference implementations: paimon-vector/.../NativeVectorGlobalIndexerFactory.java, NativeVectorGlobalIndexWriter.java, and paimon-python/pypaimon/globalindex/vindex/vindex_vector_index_writer.py in Apache Paimon master.

Paired PyPaimon integration: apache/paimon#10488 (draft until this change reaches Rust main).

Verification

• cargo test -p paimon --lib vindex --no-fail-fast: 223 passed, 1 previously ignored.
• cargo test -p paimon --features fulltext --lib index_build --no-fail-fast: 150 passed, 2 previously ignored.
• cargo test -p paimon --features fulltext --lib pk_vector --no-fail-fast: 160 passed.
• cargo test -p paimon-datafusion --features fulltext --test procedures index --no-fail-fast: 27 passed.
• cargo +1.98.0 clippy -p paimon -p pypaimon_rust --all-targets --features paimon/fulltext -- -D warnings: passed.
• cargo fmt --all -- --check and git diff --check: passed.
• Paired PyPaimon REST suites: 205 passed and 19 subtests passed, including all five vindex families, both committers, sparse original IDs queried by both readers, all-NULL/incremental shards, default/tuning options, and byte-for-byte comparisons with the existing Python writer.
• A 262,144-row regression places the only NULL in the granule plan's later read, verifies private training is discarded without an index output, then verifies sparse preparation retains logical coverage.

API and storage

Uses the existing common global-index builder and commit-message APIs. Existing index wire formats and DEIX metadata remain unchanged. Returned/submitted messages retain the existing commit ownership rules; cleanup is limited to failed private preparation.

[leaves12138]

Reviewed 94a8eb0d with apache/paimon#10488 at 971cda9c. No blocking findings.

The sparse extraction preserves original relative row IDs while keeping logical source coverage in manifest row_count/ranges. Training samples the compacted non-null vector stream; automatic IVF sizing and PQ budget validation use the real valid-vector cardinality. All-null shards avoid native training, output files and commits. The granule-to-full-spill transition keeps preparation private and drains workers before rereading the pinned snapshot. Java option mapping, field/build precedence, fixed VECTOR dimensions and the existing PK query dimension aliases are preserved.

Independent validation with Rust 1.94.1, Python 3.10, PyArrow 19.0.1 and a freshly rebuilt extension:

• Full core with full-text enabled: 4,018 passed, 6 existing ignores; cargo fmt passed.
• Related filtered runs: vindex 223 passed/1 ignored; index_build 150 passed/2 ignored; pk_vector 160 passed. These overlap the full-core run.
• DataFusion index procedures: 27 passed.
• Python bindings: 417 passed with the existing Native-generated smoke warehouse.
• Paired REST/index-build suites: 205 passed and 19 subtests passed; 55 Native plans exercised.
• Seven related vector/index/search suites: 225 passed and 59 subtests passed; 49 Native plans exercised.
• Additional reviewer tests: 30 passed across all five vindex families for fixed-size nullable VECTOR shards, dimension precedence, indexed-column updates to NULL with both committers, retained files after conflicts, and cleanup of earlier private outputs when a later shard is invalid.

Byte-for-byte full-spill comparisons against the existing Python writer passed, and both readers returned original sparse IDs. The full-core run includes the late-NULL granule regression.

Scope: Java alignment was checked against source plus the existing Python writer's byte contract, not a Java engine/production-storage interoperability run. As documented, global-index.build.parallelism is validated but native shard preparation remains sequential; this is not a concurrency/performance validation. The paired Python PR should retain its dependency sequencing until this change reaches Rust main.