Repository navigation
[python] Align INT/BIGINT and timestamp evolution with Java - #10508
Conversation
leaves12138
left a comment
There was a problem hiding this comment.
I found one blocking gap in the Java parity coverage.
Correction: INT -> TIMESTAMP was already unsupported by the Python baseline, and this PR explicitly limits its scope to existing Python-supported conversions. The reported INT gap is not introduced by this PR and should not block it.
leaves12138
left a comment
There was a problem hiding this comment.
Reviewed the current head and the paired Rust change against Java CastExecutors. The scoped BIGINT/timestamp conversions handle epoch-second units, pre-epoch flooring, LTZ epoch-day narrowing, signed-long multiplication wrapping, nested NULLs and projections consistently. Local validation: 44 Python cast tests passed; 7,105 boundary/random-value comparisons against Java passed, with identical results under UTC and Asia/Shanghai JVM time zones. The paired Rust head passed 521 Arrow tests (1 ignored) and both schema-evolution integration tests locally. My earlier INT-to-TIMESTAMP change request was dismissed and its thread resolved: that path was already unsupported in Python and is explicitly outside this PR scope. No blocking issues found in the changes. Please rebuild/rerun Native CI against upstream main containing merged apache/paimon-rust#1107 before merging this draft.
What is the purpose of the change?
Align PyPaimon numeric/timestamp schema-evolution rules with Java and verify them through Native reads. The existing Arrow cast uses physical ticks instead of epoch seconds for BIGINT paths. INT to ordinary TIMESTAMP is also rejected despite being supported by Java and Rust.
Rust prerequisite apache/paimon-rust#1107 has merged. Native CI builds paimon-rust from upstream
main.Brief change log
Verifying this change
42001ms → 42001, where Java and the corrected Python path return42.git diff --checkand Python 3.6 grammar parsing passed.CastExecutorTest#testTimestampToNumeric+testTimestampToNumericPreEpoch: 2 passed;testNumericToTimestamp: 1 passed. Direct Java CastExecutors calls confirmed overflow outputs and 20 INTEGER-to-TIMESTAMP golden values.The shared conversion and validation rules follow Java CastExecutors. No new public Python API or dependency is introduced.
Native CI follow-up
The latest Native run had 4 failures and 10,950 passing tests. The failures were strict exception-type assertions for Data Evolution deletion with deletion vectors disabled. Rust reports an unsupported operation as
NotImplementedError, matching JavaDataEvolutionDeleteSink.validateTable, while the Python delete validator and older assertions usedValueError.NotImplementedErrorexplicitly in batch, stream and Native predicate-delete regressions.ValueErrorfor invalid row IDs.git diff --checkpassed.