Iceberg updates for REST catalog and wire format compatible modes #1238
Conversation
✅ Deploy Preview for redpanda-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the 📝 WalkthroughWalkthroughThe changes update the Redpanda documentation regarding Iceberg integration. A new cluster property, Sequence Diagram(s)sequenceDiagram
participant User
participant Redpanda Docs
participant Redpanda Cluster
User->>Redpanda Docs: Consults Iceberg catalog and mode documentation
Redpanda Docs-->>User: Provides updated compatibility notes and catalog support matrix
User->>Redpanda Cluster: Configures iceberg_rest_catalog_base_location (if needed)
Redpanda Cluster-->>User: Applies property after restart (if required)
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Possibly related PRs
Suggested reviewers
✨ Finishing Touches🧪 Generate unit tests
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🔭 Outside diff range comments (2)
modules/manage/pages/iceberg/choose-iceberg-mode.adoc (1)
16-17: JSON Schema support statements contradict each otherLine 16 explicitly says JSON Schema is not supported for Iceberg topics, but the new note (lines 52-53) claims
value_schema_latestwith a JSON schema is supported starting in 25.2. Readers now have no clear guidance.Action options:
- NOTE: The JSON Schema format is not supported for Iceberg topics. If your topic data is in JSON, use the `key_value` mode. + NOTE: Prior to Redpanda 25.2, the JSON Schema format is not supported for Iceberg topics. Starting with 25.2, JSON Schema is supported only when the topic runs in `value_schema_latest` mode and producers use Redpanda Console. For any other mode, continue using `key_value`.or, if JSON Schema is still unsupported outside Console use-case, clarify that explicitly and keep the warning.
Please reconcile before publication to avoid user confusion.
Also applies to: 48-56
modules/manage/partials/iceberg/use-iceberg-catalogs.adoc (1)
60-66: Document the newiceberg_rest_catalog_base_locationproperty.The PR description highlights a new cluster property (
iceberg_rest_catalog_base_location), but it is missing from the “Set cluster properties” list. Omitting it makes the feature undiscoverable and breaks parity withcluster-properties.adoc.* config_ref:iceberg_catalog_type,true,properties/cluster-properties[`iceberg_catalog_type`]: `rest` * config_ref:iceberg_rest_catalog_endpoint,true,properties/cluster-properties[`iceberg_rest_catalog_endpoint`]: The endpoint URL for your Iceberg catalog. You either manage this directly, or you have this managed by an external catalog service. +* config_ref:iceberg_rest_catalog_base_location,true,properties/cluster-properties[`iceberg_rest_catalog_base_location`]: The base URI used by the REST catalog to resolve relative table locations. Set this when your catalog service requires an explicit warehouse-style prefix. NOTE: You must set `iceberg_rest_catalog_endpoint` at the same time that you set `iceberg_catalog_type` to `rest`.
🧹 Nitpick comments (2)
modules/manage/partials/iceberg/about-iceberg-topics.adoc (1)
138-145: Avoid duplicating full mode descriptions right after the cross-referenceThe sentence at 138 already links to
choose-iceberg-mode.adoc, yet the next block repeats the four-mode list in detail. This re-introduces the duplication the cross-reference was meant to remove and increases maintenance overhead.Consider trimming to a terse pointer:
- . Configure `redpanda.iceberg.mode` for the topic. You can choose one of the following xref:manage:iceberg/choose-iceberg-mode.adoc[Iceberg modes]: - + - -- - * `key_value`: Creates an Iceberg table using a simple schema, consisting of two columns, one for the record metadata including the key, and another binary column for the record's value. - * `value_schema_id_prefix`: Creates an Iceberg table whose structure matches the Redpanda schema for this topic, with columns corresponding to each field. You must register a schema in the Schema Registry (see next step), and producers must write to the topic using the Schema Registry wire format. - * `value_schema_latest`: Creates an Iceberg table whose structure matches the latest schema registered for the subject in the Schema Registry. - * `disabled` (default): Disables writing to an Iceberg table for this topic. - -- + . Configure `redpanda.iceberg.mode` for the topic. See the available xref:manage:iceberg/choose-iceberg-mode.adoc[Iceberg modes].This keeps a single authoritative source for mode details.
modules/reference/pages/properties/cluster-properties.adoc (1)
2093-2110: Add a “Related topics” block for cross-doc consistencyAll other Iceberg REST-catalog properties (
…_endpoint,…_token, etc.) finish with a Related topics section that links back touse-iceberg-catalogs.adoc. Adding the same here keeps navigation consistent and avoids an orphaned property.*Default:* `null` +**Related topics**: + +- xref:manage:iceberg/use-iceberg-catalogs.adoc[] --- // end::iceberg_rest_catalog_base_location[]
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
modules/manage/pages/iceberg/choose-iceberg-mode.adoc(1 hunks)modules/manage/partials/iceberg/about-iceberg-topics.adoc(1 hunks)modules/manage/partials/iceberg/use-iceberg-catalogs.adoc(2 hunks)modules/reference/pages/properties/cluster-properties.adoc(1 hunks)
🧰 Additional context used
🧠 Learnings (2)
📓 Common learnings
Learnt from: Feediver1
PR: redpanda-data/docs#1153
File: modules/reference/pages/properties/topic-properties.adoc:45-50
Timestamp: 2025-07-16T19:33:20.420Z
Learning: In the Redpanda documentation, topic property cross-references like <<max.compaction.lag.ms>> and <<min.compaction.lag.ms>> require corresponding property definition sections with anchors like [[maxcompactionlagms]] and [[mincompactionlagms]] to prevent broken links.
Learnt from: paulohtb6
PR: redpanda-data/docs#0
File: :0-0
Timestamp: 2025-07-15T20:38:27.458Z
Learning: In Redpanda documentation, "Redpanda Data" refers to the company name, while "Redpanda" refers to the product name. These terms should be used appropriately based on context.
Learnt from: Feediver1
PR: redpanda-data/docs#1153
File: antora.yml:3-5
Timestamp: 2025-07-14T19:28:43.296Z
Learning: In Redpanda docs, during beta releases, the version metadata may intentionally show inconsistencies where the header displays the beta version (e.g., 25.2 Beta) while internal attributes like full-version, latest-redpanda-tag, operator-beta-tag still reference the stable version (e.g., 25.1). This is resolved during the GA merge process when all version references are synchronized.
modules/manage/partials/iceberg/about-iceberg-topics.adoc (2)
Learnt from: Feediver1
PR: #1153
File: modules/reference/pages/properties/topic-properties.adoc:45-50
Timestamp: 2025-07-16T19:33:20.420Z
Learning: In the Redpanda documentation, topic property cross-references like <<max.compaction.lag.ms>> and <<min.compaction.lag.ms>> require corresponding property definition sections with anchors like [[maxcompactionlagms]] and [[mincompactionlagms]] to prevent broken links.
Learnt from: kbatuigas
PR: #1113
File: modules/manage/partials/iceberg/use-iceberg-catalogs.adoc:100-107
Timestamp: 2025-05-07T01:06:00.937Z
Learning: In AsciiDoc documentation for Redpanda, the syntax + and -- around content blocks within a [tabs] section are valid AsciiDoc formatting elements for tabbed content. The + after a tab name (like rpk::) indicates that the following block belongs to that tab, and the -- markers enclose the content for that tab. These are not diff artifacts and should not be removed.
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: Redirect rules - redpanda-docs-preview
- GitHub Check: Header rules - redpanda-docs-preview
- GitHub Check: Pages changed - redpanda-docs-preview
|
|
||
| *Type:* string | ||
|
|
||
| *Default:* `null` |
There was a problem hiding this comment.
@wdberkeley @alenkacz does this default also apply to Cloud?
|
|
||
| [NOTE] | ||
| ==== | ||
| The following modes are compatible with producing to an Iceberg topic using Redpanda Console: |
There was a problem hiding this comment.
@rockwotj mind signing off on this? Thank you
There was a problem hiding this comment.
Yes this information is correct
There was a problem hiding this comment.
@rockwotj did we confirm its impossible to get Console to product using the wire format (whereas rpk always does!) cc @r-vasquez
There was a problem hiding this comment.
Console only produces raw values, it does no encoding, so no support for protobuf/avro with or without schema registry.
There was a problem hiding this comment.
It would be cool to support json or textproto for console testing (with or without wire format)
| The following modes are compatible with producing to an Iceberg topic using Redpanda Console: | ||
| - `key_value` | ||
| - Starting in version 25.2, `value_schema_latest` with a JSON schema |
There was a problem hiding this comment.
Note: The PR to add JSON Schema support is open and will be merged when 25.2 is GA.
| |=== | ||
| endif::[] | ||
|
|
||
| Other REST catalogs such as Dremio Nessie, Apache Polaris, and the Apache reference implementation have been tested but are not regularly verified. For more information, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda Support^]. |
There was a problem hiding this comment.
@wzzzrd86 mind signing off on this? Thank you
| |Databricks Unity Catalog |Tested | ||
| |Snowflake Open Catalog |Tested | ||
| |Dremio Nessie |Tested | ||
| |AWS Glue Data Catalog | Beta |
There was a problem hiding this comment.
@kbatuigas a nit, but maybe we list the 'supported' ones first?
|
|
||
| *Type:* string | ||
|
|
||
| *Default:* `null` |
|
|
||
| [NOTE] | ||
| ==== | ||
| The following modes are compatible with producing to an Iceberg topic using Redpanda Console: |
There was a problem hiding this comment.
@rockwotj did we confirm its impossible to get Console to product using the wire format (whereas rpk always does!) cc @r-vasquez
Description
This pull request introduces updates to the Redpanda Iceberg integration documentation, focusing on new features, compatibility updates, and catalog support. Key changes include the addition of supported Iceberg modes, updates to the catalog compatibility matrix, and a new configuration property for Iceberg REST catalogs.
Updates to Iceberg mode and configuration:
modules/manage/pages/iceberg/choose-iceberg-mode.adocspecifying that thevalue_schema_latestmode with JSON schema is supported starting in version 25.2, along with a warning about potential record failures.modules/manage/partials/iceberg/about-iceberg-topics.adocto include a cross-reference to the Iceberg modes documentation and removed an unnecessary "See also" link. [1] [2]Updates to Iceberg catalog support:
modules/manage/partials/iceberg/use-iceberg-catalogs.adocto include AWS Glue Data Catalog and updated the status for Azure and GCP.New Iceberg REST catalog property:
iceberg_rest_catalog_base_location, inmodules/reference/pages/properties/cluster-properties.adoc, which allows users to set the base URI for the Iceberg REST catalog. This property is available starting in Redpanda version 25.1.7.Resolves https://redpandadata.atlassian.net/browse/
Review deadline: 29 July
Page previews
Cloud: Use Iceberg Catalogs
Self-managed: Use Iceberg Catalogs
Cluster config reference:
iceberg_rest_catalog_base_locationSupported Iceberg modes
Checks