[#10674] docs(iceberg): Add design doc for multi-table atomic commit in Iceberg REST catalog - #12195
Open
laserninja wants to merge 1 commit into
Open
Conversation
…ommit
Adds a design document for the Iceberg REST catalog commitTransaction
endpoint (POST /v1/{prefix}/transactions/commit), requested during review
of apache#10675.
The document proposes backend-capability dispatch: implement a genuinely
atomic commit path for each backend that can support one (batched CAS in a
single DB transaction for JDBC, raw request forwarding for REST), and
return 501 on backends that cannot (HIVE, CUSTOM), gating the
/v1/config advertisement of V1_COMMIT_TRANSACTION on that capability.
Also covers per-table authorization to close the privilege-escalation gap
identified in review, and documents the orphaned-metadata and
lost-response semantics.
Code Coverage Report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Adds a design document for the Iceberg REST catalog
commitTransactionendpoint(
POST /v1/{prefix}/transactions/commit):design-docs/iceberg-multi-table-atomic-commit-design.md.No production code changes.
The document proposes backend-capability dispatch rather than one uniform best-effort
implementation:
JDBCConnectionin one DB transactionRESTMEMORYHIVE501; HMS Thrift has no multi-table transactional alterCUSTOMCatalogopts inEndpoint.V1_COMMIT_TRANSACTIONis advertised in/v1/configif and only if the resolved backendcan commit atomically, following the existing precedent where
getEndpoints(supportsViewOperations)already varies the advertised set by catalog capability. This means a client never sees the endpoint
advertised unless the guarantee behind it is real.
The document also covers:
@AuthorizationExpressioninterceptor resolves entities from path parameters, but this endpointcarries its tables in the request body, so authorization is done programmatically via
MetadataAuthzHelper.checkAccessagainst the same expressionupdateTableuses. The expression isextracted into a shared constant so the two paths cannot drift.
validate-then-commitapproach from [#10674] feat(iceberg): implement commitTransaction REST endpoint for Iceberg REST catalog #10675 and aPolaris-style Gravitino-owned pointer store, with concrete reasons.
Why are the changes needed?
POST /v1/{prefix}/transactions/commitis part of the Iceberg REST Catalog spec and Gravitino doesnot implement it, so engines cannot perform cross-table atomic operations against Gravitino.
The first implementation attempt (#10675) was found not to deliver atomic semantics: the commit phase
remained N independent compare-and-swap operations, so a failure partway through left a partial state.
Reviewers concluded that multi-table atomicity needs a design document before implementation, and
asked for one:
The design work also answers the question raised in
#10675 (comment) about which backends can
achieve true CAS: the capability differs per backend, and that asymmetry is what drives the proposed
design.
Fix: #10674
Does this PR introduce any user-facing change?
No. Documentation only.
The design it describes would introduce a new REST endpoint and a new entry in the
/v1/configendpointslist, but no code is changed in this PR.How was this patch tested?
N/A, documentation only.
Claims about backend behaviour in the document were verified against the Iceberg 1.11.0 artifacts
this repo depends on rather than from memory, including
JdbcUtil's commit SQL and discriminatorcolumn,
HiveTableOperations.doCommit's per-tablealter_tablepath,TableCommit.create'srequirement derivation, and the visibility of
ResourcePaths.commitTransaction()andRESTClient.post.