Skip to content

[python] Align INT/BIGINT and timestamp evolution with Java - #10508

Merged
JingsongLi merged 3 commits into
apache:masterfrom
JingsongLi:codex/native-temporal-evolution
Oct 12, 2026
Merged

JingsongLi merged 3 commits into
apache:masterfrom
JingsongLi:codex/native-temporal-evolution

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

What is the purpose of the change?

Align PyPaimon numeric/timestamp schema-evolution rules with Java and verify them through Native reads. The existing Arrow cast uses physical ticks instead of epoch seconds for BIGINT paths. INT to ordinary TIMESTAMP is also rejected despite being supported by Java and Rust.

Rust prerequisite apache/paimon-rust#1107 has merged. Native CI builds paimon-rust from upstream main.

Brief change log

  • Use epoch seconds for TIMESTAMP/TIMESTAMP_LTZ → BIGINT, including floor division before the epoch and Java's LTZ local-date-time round trip.
  • Use Java's signed-long intermediate multiplication for INTEGER/BIGINT → ordinary TIMESTAMP and retain NULLs for Arrow storage overflow.
  • Apply these conversions in the shared schema-evolution helper used by scalar, recursive nested and leaf-projection readers. ALTER executable-cast validation probes that same executor, including INT → TIMESTAMP conversions with no direct Arrow kernel.
  • Add 38 conversion unit cases for sliced/chunked arrays, NULLs, Java boundary values and Arrow units, plus ALTER executable-cast validation coverage.
  • Add 62 REST Native parity cases for append, primary-key and data-evolution tables, covering mixed schema versions, single-value file statistics, predicate-only fields, NULLs, ROW/ARRAY/MAP evolution and nested/MAP key projections. Native checks explicitly forbid Python read/plan fallback.

Verifying this change

  • Reproduced the old Native result 42001ms → 42001, where Java and the corrected Python path return 42.
  • Rebuilt the Native extension using the dependent Rust branch.
  • Conversion, REST Native and nested schema-evolution suites after review fixes: 234 passed, 8 subtests passed with Native plan/read/write/commit enabled (382 Native plans/reads, 244 Native writes, 207 Native commits).
  • Additional scalar/PK read and MAP/nested projection regressions, with one REST case to exercise the Native commit gate: 73 passed, 21 subtests passed. In total, 306 distinct tests passed.
  • INT and BIGINT nested epoch/MAP key projection checks passed across append, primary-key and data-evolution tables.
  • Changed-file Flake8, git diff --check and Python 3.6 grammar parsing passed.
  • Java CastExecutorTest#testTimestampToNumeric+testTimestampToNumericPreEpoch: 2 passed; testNumericToTimestamp: 1 passed. Direct Java CastExecutors calls confirmed overflow outputs and 20 INTEGER-to-TIMESTAMP golden values.

The shared conversion and validation rules follow Java CastExecutors. No new public Python API or dependency is introduced.

Native CI follow-up

The latest Native run had 4 failures and 10,950 passing tests. The failures were strict exception-type assertions for Data Evolution deletion with deletion vectors disabled. Rust reports an unsupported operation as NotImplementedError, matching Java DataEvolutionDeleteSink.validateTable, while the Python delete validator and older assertions used ValueError.

  • Align Python's unsupported deletion capability checks with Java and Native.
  • Assert NotImplementedError explicitly in batch, stream and Native predicate-delete regressions.
  • Cover missing data evolution, row tracking and deletion vectors, and retain ValueError for invalid row IDs.
  • Focused Python regressions: 10 passed.
  • Native update/delete/deletion-vector suites: 178 passed, 20 subtests passed; 145 Native plans, 161 reads, 270 writes and 69 commits exercised.
  • Changed-file Flake8, Python 3.6 grammar and git diff --check passed.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found one blocking gap in the Java parity coverage.

Comment thread paimon-python/pypaimon/casting/schema_evolution.py Outdated
@leaves12138
leaves12138 dismissed their stale review October 12, 2026 03:00

Correction: INT -> TIMESTAMP was already unsupported by the Python baseline, and this PR explicitly limits its scope to existing Python-supported conversions. The reported INT gap is not introduced by this PR and should not block it.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the current head and the paired Rust change against Java CastExecutors. The scoped BIGINT/timestamp conversions handle epoch-second units, pre-epoch flooring, LTZ epoch-day narrowing, signed-long multiplication wrapping, nested NULLs and projections consistently. Local validation: 44 Python cast tests passed; 7,105 boundary/random-value comparisons against Java passed, with identical results under UTC and Asia/Shanghai JVM time zones. The paired Rust head passed 521 Arrow tests (1 ignored) and both schema-evolution integration tests locally. My earlier INT-to-TIMESTAMP change request was dismissed and its thread resolved: that path was already unsupported in Python and is explicitly outside this PR scope. No blocking issues found in the changes. Please rebuild/rerun Native CI against upstream main containing merged apache/paimon-rust#1107 before merging this draft.

@JingsongLi JingsongLi changed the title [python] Align BIGINT and timestamp evolution with Java [python] Align INT/BIGINT and timestamp evolution with Java Oct 12, 2026
@JingsongLi
JingsongLi marked this pull request as ready for review October 12, 2026 04:09
@JingsongLi
JingsongLi merged commit d767698 into apache:master Oct 12, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants