[EPIC] Add Iceberg REST soft-delete lifecycle and recovery APIs

[nevzheng]

Describe the proposal

This is an Iceberg REST API-focused child epic of #12219 and depends on the deletion storage foundation in #12220.

Add opt-in soft deletion and recovery for Iceberg REST tables while preserving existing behavior when soft deletion is disabled.

DELETE behavior

The existing Iceberg REST DELETE endpoint remains the entry point for deleting a table.

When the deletion lifecycle applies, DELETE should:

• Create one immutable deletion ID.
• Insert its active deletion action and mark the existing table row as deleted in one relational transaction.
• Reserve the table name by retaining the original row.
• Return 204 No Content only after the deletion metadata is durable.
• Never delete Iceberg files on the request thread.

A repeated DELETE against the same active deletion should be idempotent: it returns success without creating another deletion ID or changing the original deletion time.

Normal table list and load operations must continue to hide deleted tables. Creating or registering another table under a reserved name must fail until the original deletion is restored or purged.

Deleted-table discovery

Reuse the existing Iceberg REST table resources rather than introducing specialized deletion-discovery routes:

• Omitting deleted, or using deleted=false, returns live tables.
• Using deleted=true returns deleted tables only.
• Existing pageToken and pageSize behavior applies to deleted-table listings.
• Loading a deleted table joins its retained table row with the current deletion action.
• Deleted-table responses include the deletion ID, lifecycle state, deletion time, retention deadline, recoverability, and a strong ETag derived from the deleted representation.
• Responses must not expose Iceberg metadata locations, FileIO configuration, credentials, or storage secrets.
• Deleted-table responses should prevent unsafe caching.

UNDROP behavior

Add an explicit action endpoint:

POST /management/v1/{prefix}/namespaces/{namespace}/tables/{table}/undrop

The request contains the exact deletionId and must include its strong ETag in If-Match.

The deletion ID identifies the deletion generation. The ETag validates that the client is acting on its current representation.

A successful UNDROP must:

• Verify that the exact deletion remains recoverable.
• Reactivate the original table_meta row in one relational transaction.
• Preserve the original table identity and table-owned metadata linked through table_id.
• Clear the table's active deletion pointer and consume the active deletion action.
• Return the ordinary live Iceberg table response and live-table ETag.

UNDROP is an action, not a generic metadata PATCH. It does not create a replacement table or perform an unregister/register saga.

An UNDROP request must not affect an unrelated live table or a later deletion using the same name. Repeating UNDROP after the row is already live has no further effect and is reported as an invalid current target rather than restoring anything again.

Concurrency and errors

DELETE, create, discovery, and UNDROP must use the exact deletion ID and fine-grained relational row coordination. The implementation must not use table-wide database locks.

The API should distinguish malformed requests, missing preconditions, stale ETags, conflicting current state, expired or purge-boundary deletions, missing objects, and authorization failures with consistent HTTP status codes.

Unauthorized callers must not be able to use deleted-table APIs to discover metadata they could not otherwise access.

Testing

Add tests covering:

• Soft deletion enabled and disabled.
• Retention boundaries, including zero retention.
• DELETE retry behavior.
• Live-only and deleted-only list and load behavior.
• Deleted-table pagination.
• Strong ETag generation and conditional requests.
• Missing, malformed, weak, wildcard, and stale If-Match values.
• Successful UNDROP and repeated requests against an already-live row.
• Same-name creation conflicts.
• Concurrent lifecycle changes.
• Authorization and concealment of inaccessible deletions.
• Exclusion of credentials and internal cleanup data from API responses.

This epic does not implement garbage-collector selection, physical Iceberg file deletion, purge workers, job progress APIs, or large-scale cleanup testing. Those capabilities remain in the pending purge and validation epics.

Task list

• Define implementation subtasks and reviewable PR boundaries.
• [#12221] feat(iceberg): Add retained table deletion coordinator #12225 — Review 1/5: soft-delete configuration and retained deletion coordinator.
• [#12221] feat(iceberg): Wire retained table delete and cache coherence #12228 — Review 2/5: existing DELETE endpoint wiring and cache coherence.
• [#12221] feat(iceberg): Add retained deletion read model #12229 — Review 3/5: joined retained-deletion read model and strong ETag.
• [#12221] feat(iceberg): Expose retained table deletions #12226 — Review 4/5: deleted-table reads through the existing table API.
• [#12221] feat(iceberg): Add conditional table undrop #12227 — Review 5/5: conditional UNDROP of the original table row.
• Purge and durable job-system epic — pending.
• Integration and scale-validation epic — pending.

Review order: #12222 → #12223 → #12225 → #12228 → #12229 → #12226 → #12227.