Skip to content

[core] Unify catalog locks for snapshot commits - #10489

Closed
JingsongLi wants to merge 7 commits into
apache:masterfrom
JingsongLi:codex/rest-commit-lock-on-retry
Closed

JingsongLi wants to merge 7 commits into
apache:masterfrom
JingsongLi:codex/rest-commit-lock-on-retry

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Configured catalog locks should protect reading the current head, validating changes, preparing manifests and publishing each snapshot. This change uses one table-bound Lock from Catalog.createLock for every attempt, including the first, with separate scopes for APPEND and COMPACT publications. Post-commit callbacks run after release. It removes the retry-only option and the CommitAttempt / CatalogCommitLock interfaces while preserving the public CatalogLockFactory SPI and direct snapshotCommit() publication-lock behavior.

The REST protocol and minimal public contract are proposed separately in #10491. This branch currently contains that foundation as well as the writer integration; it can be rebased after the foundation lands.

Hive locks now heartbeat and verify ownership before publication. JDBC leases renew and release by an internal owner, reject expired renewal, and keep renewal deadlines monotonic. Existing JDBC lock tables receive a nullable lock_owner column on catalog initialization, requiring ALTER permission. All JDBC writers should upgrade before relying on owner-checked release after expiry. Filesystem locking does not add a storage fencing token.

REST automatic conflict rollback carries the table UUID and commit user so a server can atomically validate the expected head and lease, roll back and clear the lease. Snapshot/file conflict checks remain required; leases do not make stale file changes valid.

Validation:

  • JDK 8 and JDK 11: all 177 tests in CompactReassignReuseTest, DataEvolutionRowIdReassignerTest, CatalogEnvironmentTest and FileStoreCommitTest passed with normal Maven checks. Fault injection now overrides the publication-lock factory used by the writer, retaining all conflict and lost-response assertions.
  • JDK 8: 41 request compatibility tests, 198 core/REST/JDBC/commit tests and 6 Hive lease tests passed, with normal Maven checks enabled.
  • Coverage includes lock-before-head ordering, every-attempt acquisition, separate APPEND/COMPACT scopes, callbacks after release, invalid leases blocking publication, legacy snapshots without UUID, direct publisher compatibility, JDBC owner takeover/expiry/schema upgrade, and interleaved heartbeat responses.
  • REST OpenAPI validation passed for 62 catalog and 10 management operations.
  • MySQL/PostgreSQL container integration was not run locally because the Docker daemon is unavailable.

@JingsongLi JingsongLi changed the title [core] Support REST commit leases on retry [core] Unify catalog locks for snapshot commits Oct 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant