Partition evolution supported in Unity Catalog #1163
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the 📝 WalkthroughWalkthroughThe documentation was updated in three areas related to Iceberg integration. The section on Databricks Unity Catalog was revised to remove references to the lack of support for partition evolution and the cluster property Sequence Diagram(s)No sequence diagrams are generated as the changes are limited to documentation updates and do not introduce or modify control flows or features. Assessment against linked issues
Assessment against linked issues: Out-of-scope changesNo out-of-scope changes were found. All documentation updates align with the linked issue's objectives. Possibly related PRs
Suggested reviewers
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
✅ Deploy Preview for redpanda-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
modules/reference/pages/properties/cluster-properties.adoc (1)
2069-2073: Good cloud-specific recommendation
The newifdef::env-cloud[]block correctly highlights that OAuth2 is preferred for REST catalog authentication in cloud environments.Optionally, consider appending a xref to the secret creation guide (e.g.
manage/partials/iceberg/use-iceberg-catalogs.adoc) so users can jump directly from this recommendation to configuringiceberg_rest_catalog_client_idandiceberg_rest_catalog_client_secret.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc(2 hunks)modules/manage/partials/iceberg/use-iceberg-catalogs.adoc(2 hunks)modules/reference/pages/properties/cluster-properties.adoc(1 hunks)
🔇 Additional comments (8)
modules/manage/partials/iceberg/use-iceberg-catalogs.adoc (7)
63-67: Add cloud-specific recommendation for oauth2 authentication mode.Introducing an
ifdef::env-cloud[]block to recommend usingoauth2in cloud environments aligns docs with best practices. Confirm that this aligns with the corresponding property docs incluster-properties.adoc.
68-75: Remove AsciiDoc block delimiters for clarity.The removal of
--separators refines the list formatting, but verify that the nested list items render correctly in all outputs.
76-77: Add caution for bearer token usage.The new note clarifies that bearer tokens are not refreshed, highlighting security considerations for ad hoc or testing scenarios.
112-114: Ensure correct AsciiDoc tab block formatting.A
+was added before the block delimiter; validate that the[tabs]and block syntax render as intended in the generated documentation.
119-123: Expand placeholders guidance for secret creation.Providing explicit instructions for
<secret-name>and<secret-value>enhances usability and reduces misconfiguration risk.
127-133: Streamline Cloud API instructions.Added
+markers and removed separators around the Cloud API section; confirm the resulting layout maintains proper indentation and readability.
140-150: Detail secret payload parameters.Listing all required placeholders and their patterns in this section ensures users supply valid values when creating secrets via the Cloud API.
modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc (1)
30-41: Reposition unsupported data types listing.The table of unsupported data types has been moved to appear immediately after the Limitations heading, improving the document’s flow and clarity.
|
|
||
| == Limitations | ||
|
|
||
| The following data types are not currently supported for managed Iceberg tables: |
There was a problem hiding this comment.
@nvartolomei this looks correct me to based on recent slack convos but can you take a quick 👀 , please?
kbatuigas
force-pushed
the
DOC-1446-unity-catalog-supports-partition-evolution-with-ex
branch
from
81bdd3f to
9d21596
Compare
Co-authored-by: Michele Cyran <michele@redpanda.com>
kbatuigas
deleted the
DOC-1446-unity-catalog-supports-partition-evolution-with-ex
branch
Co-authored-by: Michele Cyran <michele@redpanda.com>
Description
This pull request introduces documentation updates for Redpanda's Iceberg integration, focusing on partitioning, authentication modes, dead-letter queues, and topic properties. The updates aim to improve clarity and provide additional guidance for users configuring Iceberg-related features.
Iceberg Partitioning and Configuration Updates:
Authentication Recommendations:
oauth2as the preferred authentication mode for REST catalogs in cloud environments. (modules/manage/partials/iceberg/use-iceberg-catalogs.adoc, [1];modules/reference/pages/properties/cluster-properties.adoc, [2]Dead-Letter Queue (DLQ) Configuration:
redpanda.iceberg.invalid.record.actiontopic property, explaining options for handling invalid records with a dead-letter queue (dlq_table) or dropping them (drop). (modules/reference/pages/properties/topic-properties.adoc, modules/reference/pages/properties/topic-properties.adocR564-R580)Topic Refresh Lag Configuration:
redpanda.iceberg.target.lag.msproperty, which controls the frequency of data refreshes in Iceberg tables, with a default value of 60,000 milliseconds. (modules/reference/pages/properties/topic-properties.adoc, modules/reference/pages/properties/topic-properties.adocL571-R597)Cross-Reference Improvements:
modules/reference/pages/properties/topic-properties.adoc, [1]; [2]Resolves https://redpandadata.atlassian.net/browse/
Review deadline:
Page previews
Query Iceberg Topics with Databricks Unity Catalog > Limitations
Topic Configuration Properties > redpanda.iceberg.target.lag.ms
Cloud > Use Iceberg Catalogs > Connect to REST Catalog > Set cluster properties
Cloud > Cluster Properties > iceberg_rest_catalog_authentication_mode
Checks