Partition evolution supported in Unity Catalog (#1163)

Co-authored-by: Michele Cyran <michele@redpanda.com>

Author: 2 people

modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc

@@ -27,13 +27,9 @@ endif::[]
 * Workspace admin privileges to complete the steps to create a Unity Catalog storage credential and external location that connects your cluster's Tiered Storage bucket to Databricks.
 
 == Limitations
+ 
+The following data types are not currently supported for managed Iceberg tables:
 
-* Databricks Managed Iceberg tables do not currently support partition evolution. If you define a xref:manage:iceberg/about-iceberg-topics.adoc#use-custom-partitioning[custom partitioning] scheme for an Iceberg topic, you must not change it later. 
-+
-The cluster property config_ref:iceberg_default_partition_spec,true,properties/cluster-properties[`iceberg_default_partition_spec`] defines the cluster-wide partitioning scheme, by default configured to `(hour(redpanda.timestamp))`. To use a different cluster-wide default, configure this property before creating any Iceberg-enabled topics. You must only override the default partitioning scheme on the topic level for new Iceberg topics that do not yet contain data.  
-* The following data types are not currently supported for managed Iceberg tables:
-+
---
 |===
 | Iceberg type | Equivalent Avro type
 
@@ -42,8 +38,7 @@ The cluster property config_ref:iceberg_default_partition_spec,true,properties/c
 | time | time-millis, time-micros
 
 |===
---
-+
+
 There are no limitations for Protobuf types.
 
 == Create a Unity Catalog storage credential

modules/manage/partials/iceberg/use-iceberg-catalogs.adoc

@@ -60,6 +60,11 @@ To connect to a REST catalog, set the following cluster configuration properties
 * config_ref:iceberg_catalog_type,true,properties/cluster-properties[`iceberg_catalog_type`]: `rest`
 * config_ref:iceberg_rest_catalog_endpoint,true,properties/cluster-properties[`iceberg_rest_catalog_endpoint`]: The endpoint URL for your Iceberg catalog, which you either manage directly, or is managed by an external catalog service.
 * config_ref:iceberg_rest_catalog_authentication_mode,true,properties/cluster-properties[`iceberg_rest_catalog_authentication_mode`]: The authentication mode to use for the REST catalog. Choose from `oauth2`, `bearer`, or `none` (default).
+ifdef::env-cloud[]
++
+Redpanda recommends using `oauth2`.
+
+endif::[]
 ** For `oauth2`, also configure the following properties:
 +
 --
@@ -111,6 +116,11 @@ Run the following `rpk` command:
 ----
 rpk security secret create --name <secret-name> --value <secret-value> --scopes redpanda_cluster
 ----
+
+Replace the placeholders with your own values:
+
+- `<secret-name>`: The name of the secret you want to add. The secret name is also its ID. Use only the following characters: `^[A-Z][A-Z0-9_]*$`.
+- `<secret-value>`: The value of the secret.
 --
 
 Cloud API::

modules/reference/pages/properties/cluster-properties.adoc

@@ -2066,7 +2066,11 @@ endif::[]
 // tag::iceberg_rest_catalog_authentication_mode[]
 === iceberg_rest_catalog_authentication_mode
 
-The authentication mode for client requests made to the Iceberg catalog. Choose from: `none`, `bearer`, and `oauth2`. In `bearer` mode, the token specified in `iceberg_rest_catalog_token` is used unconditionally, and no attempts are made to refresh the token. In `oauth2` mode, the credentials specified in `iceberg_rest_catalog_client_id` and `iceberg_rest_catalog_client_secret` are used to obtain a bearer token from the URI defined by `iceberg_rest_catalog_oauth2_server_uri.`.
+The authentication mode for client requests made to the Iceberg catalog. Choose from: `none`, `bearer`, and `oauth2`. In `oauth2` mode, the credentials specified in `iceberg_rest_catalog_client_id` and `iceberg_rest_catalog_client_secret` are used to obtain a bearer token from the URI defined by `iceberg_rest_catalog_oauth2_server_uri.`. In `bearer` mode, the token specified in `iceberg_rest_catalog_token` is used unconditionally, and no attempts are made to refresh the token. 
+
+ifdef::env-cloud[]
+Redpanda recommends using `oauth2`.
+endif::[]
 
 *Requires restart:* Yes
 

modules/reference/pages/properties/topic-properties.adoc

@@ -544,6 +544,7 @@ Enable the Iceberg integration for the topic. You can choose one of four modes.
 
 **Related topics**:
 
+- xref:manage:iceberg/choose-iceberg-mode.adoc[]
 - xref:manage:iceberg/about-iceberg-topics.adoc[]
 
 ---
@@ -560,6 +561,23 @@ Whether the corresponding Iceberg table is deleted upon deleting the topic.
 
 ---
 
+==== redpanda.iceberg.invalid.record.action
+
+Whether to write invalid records to a dead-letter queue (DLQ).
+
+**Default**: `dlq_table`
+
+**Values**:
+
+- `drop`: Disable the DLQ and drop invalid records.
+- `dlq_table`: Write invalid records to a separate DLQ Iceberg table.
+
+**Related topics**:
+
+- xref:manage:iceberg/about-iceberg-topics.adoc#manage-dead-letter-queue[Manage dead-letter queue]
+
+---
+
 ==== redpanda.iceberg.partition.spec
 
 The https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] specification for the Iceberg table.
@@ -568,20 +586,15 @@ The https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] specifi
 
 **Related topics**:
 
-- xref:manage:iceberg/about-iceberg-topics.adoc[]
+- xref:manage:iceberg/about-iceberg-topics.adoc#use-custom-partitioning[Use custom partitioning]
 
 ---
 
-==== redpanda.iceberg.invalid.record.action
-
-Whether to write invalid records to a dead-letter queue (DLQ).
-
-**Default**: `dlq_table`
+==== redpanda.iceberg.target.lag.ms
 
-**Values**:
+Controls how often the data in the Iceberg table is refreshed with new data from the topic. Redpanda attempts to commit all data produced to the topic within the lag target, subject to resource availability.
 
-- `drop`: Disable the DLQ and drop invalid records.
-- `dlq_table`: Write invalid records to a separate DLQ Iceberg table.
+**Default**: `60000`
 
 **Related topics**: