Skip to content

[core] Align BIGINT and timestamp schema evolution with Java - #1107

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-temporal-evolution
Oct 12, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-temporal-evolution

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

What is the purpose of the change?

Align the existing BIGINT/timestamp schema-evolution paths with Java casts. PyPaimon already reads these type changes, but Arrow's generic cast interprets timestamps as physical ticks: for example 42001ms becomes BIGINT 42001, while Java returns epoch seconds 42.

Brief change log

  • Convert TIMESTAMP and TIMESTAMP_LTZ to BIGINT using epoch seconds and floor division for pre-epoch values.
  • Match Java's LTZ Timestamp local-date-time round trip, including its int epoch day.
  • Move the existing numeric-to-ordinary-TIMESTAMP conversion into the shared evolution cast module and apply Java signed-long wrapping to the intermediate seconds * 1000 milliseconds. Preserve lenient NULLs when Arrow's target unit cannot represent the result.
  • Reuse recursive evolution and predicate/statistics paths without adding a public API or dependency.
  • Cover Arrow units, slices, NULLs, Java boundary outputs, nullable ROW/ARRAY/MAP values, and mixed old/new files with predicates and projection across append, primary-key and data-evolution tables.

Verifying this change

  • Reproduced the old timestamp-to-BIGINT unit error in both the core test and a REST Native integration test.
  • cargo test -p paimon --lib arrow::: 522 passed, 1 ignored; after adding the final nullable-container test, all 28 nested-evolution tests passed.
  • cargo test -p paimon --test schema_evolution_cast_test: 2 passed (each covers append, PK and data evolution).
  • cargo +1.98.0 fmt --all -- --check
  • cargo +1.98.0 clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings
  • Java CastExecutorTest#testTimestampToNumeric+testTimestampToNumericPreEpoch: 2 passed. Direct calls to Java CastExecutors also confirmed the long-wrapping and extreme-LTZ golden values.
  • Rebuilt the Python extension from this branch: 166 focused Python schema-evolution tests, 112 existing read/projection tests, and final MAP key projection checks passed.

The accompanying PyPaimon PR apache/paimon#10508 aligns its existing Python casts with Java and adds Native REST parity coverage. This only addresses type-change paths already supported by Python.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Rust implementation aligns the BIGINT/timestamp evolution paths with Java, including floor division, LTZ round-trip behavior, signed-long wrapping, nested values, and statistics/predicate safety. The focused Rust tests and CI are green.

@JingsongLi
JingsongLi merged commit ae858ea into apache:main Oct 12, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants