Skip to content

Commit ecbec8c

Browse files
kbatuigasFeediver1
andauthored
Add BYOVPC support for Iceberg REST catalogs in single-sourced docs (#1140)
Co-authored-by: Joyce Fee <102751339+Feediver1@users.noreply.github.com>
1 parent 1a60814 commit ecbec8c

File tree

3 files changed

+62
-4
lines changed

3 files changed

+62
-4
lines changed

modules/manage/partials/iceberg/about-iceberg-topics.adoc

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,9 @@ SUBJECT VERSION ID TYPE
150150
----
151151

152152
ifdef::env-cloud[]
153+
154+
=== Access Iceberg data
155+
153156
To query the Iceberg table, you need access to the object storage bucket or container where the Iceberg data is stored. For BYOC clusters, the bucket name and table location are as follows:
154157

155158
|===
@@ -170,6 +173,29 @@ The Redpanda cluster ID is also used as the container name (ID) and the storage
170173

171174

172175
|===
176+
177+
For Azure clusters, you must add the public IP addresses or ranges from the REST catalog service, or other clients requiring access to the Iceberg data, to your cluster's allow list. Alternatively, add subnet IDs to the allow list if the requests originate from the same Azure region.
178+
179+
For example, to add subnet IDs to the allow list through the Control Plane API xref:api:ROOT:cloud-controlplane-api.adoc#patch-/v1/clusters/-cluster.id-[`PATCH /v1/clusters/<cluster-id>`] endpoint, run:
180+
181+
[,bash]
182+
----
183+
curl -X PATCH https://api.cloud.redpanda.com/v1/clusters/<cluster-id> \
184+
-H "Content-Type: application/json" \
185+
-H "Authorization: Bearer ${RP_CLOUD_TOKEN}" \
186+
-d @- << EOF
187+
{
188+
"cloud_storage": {
189+
"azure": {
190+
"allowed_subnet_ids": [
191+
<list-of-subnet-ids>
192+
]
193+
}
194+
}
195+
}
196+
EOF
197+
----
198+
173199
endif::[]
174200

175201
As you produce records to the topic, the data also becomes available in object storage for Iceberg-compatible clients to consume. You can use the same analytical tools to xref:manage:iceberg/query-iceberg-topics.adoc[read the Iceberg topic data] in a data lake as you would for a relational database.

modules/manage/partials/iceberg/query-iceberg-topics.adoc

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,29 @@ The Redpanda cluster ID is also used as the container name (ID) and the storage
2525
| `redpanda-cloud-storage-<cluster-id>`
2626

2727
|===
28+
29+
For Azure clusters, you must add the public IP addresses or ranges from the REST catalog service, or other clients requiring access to the Iceberg data, to your cluster's allow list. Alternatively, add subnet IDs to the allow list if the requests originate from the same Azure region.
30+
31+
For example, to add subnet IDs to the allow list through the Control Plane API xref:api:ROOT:cloud-controlplane-api.adoc#patch-/v1/clusters/-cluster.id-[`PATCH /v1/clusters/<cluster-id>`] endpoint, run:
32+
33+
[,bash]
34+
----
35+
curl -X PATCH https://api.cloud.redpanda.com/v1/clusters/<cluster-id> \
36+
-H "Content-Type: application/json" \
37+
-H "Authorization: Bearer ${RP_CLOUD_TOKEN}" \
38+
-d @- << EOF
39+
{
40+
"cloud_storage": {
41+
"azure": {
42+
"allowed_subnet_ids": [
43+
<list-of-subnet-ids>
44+
]
45+
}
46+
}
47+
}
48+
EOF
49+
----
50+
2851
endif::[]
2952

3053
If your engine needs the full JSON metadata path, use the following:

modules/manage/partials/iceberg/use-iceberg-catalogs.adoc

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,6 @@ ifndef::env-cloud[:about-iceberg-doc: manage:iceberg/topic-iceberg-integration.a
33

44
To read from the Redpanda-generated xref:{about-iceberg-doc}[Iceberg table], your Iceberg-compatible client or tool needs access to the catalog to retrieve the table metadata and know the current state of the table. The catalog provides the current table metadata, which includes locations for all the table's data files. You can configure Redpanda to either connect to a REST-based catalog, or use a filesystem-based catalog.
55

6-
ifdef::env-cloud[]
7-
NOTE: The Iceberg integration for Redpanda Cloud is a beta feature. It is not supported for production deployments. To configure REST catalog authentication for use with Iceberg topics in your cloud cluster, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda support^].
8-
endif::[]
9-
106
For production deployments, Redpanda recommends using an external REST catalog to manage Iceberg metadata. This enables built-in table maintenance, safely handles multiple engines and tools accessing tables at the same time, facilitates data governance, and maximizes data discovery. However, if it is not possible to use a REST catalog, you may use the filesystem-based catalog (`object_storage` catalog type), which does not require you to maintain a separate service to access the Iceberg data. In either case, you use the catalog to load, query, or refresh the Iceberg table as you produce to the Redpanda topic. See the documentation for your query engine or Iceberg-compatible tool for specific guidance on adding the Iceberg tables to your data warehouse or lakehouse using the catalog.
117

128
After you have selected a catalog type at the cluster level and xref:{about-iceberg-doc}#enable-iceberg-integration[enabled the Iceberg integration] for a topic, you cannot switch to another catalog type.
@@ -15,6 +11,19 @@ After you have selected a catalog type at the cluster level and xref:{about-iceb
1511

1612
Connect to an Iceberg REST catalog using the standard https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml[REST API^] supported by many catalog providers. Use this catalog integration type with REST-enabled Iceberg catalog services, such as https://docs.databricks.com/en/data-governance/unity-catalog/index.html[Databricks Unity^] and https://other-docs.snowflake.com/en/opencatalog/overview[Snowflake Open Catalog^].
1713

14+
ifdef::env-cloud[]
15+
=== Prerequisites
16+
17+
For BYOVPC clusters, you must:
18+
19+
. Enable secrets management, which allows you to store and use secrets in your cluster's Iceberg catalog authentication properties.
20+
+
21+
Secrets management is enabled by default for AWS if you follow the guide to xref:get-started:cluster-types/byoc/aws/vpc-byo-aws.adoc[creating a new BYOVPC cluster]. For GCP, follow the guides to enable secrets management for a xref:get-started:cluster-types/byoc/gcp/vpc-byo-gcp.adoc[new BYOVPC cluster] or an xref:get-started:cluster-types/byoc/gcp/enable-secrets-byovpc-gcp.adoc[existing BYOVPC cluster].
22+
. Ensure that your network security settings allow egress traffic from the Redpanda network to the catalog service endpoints.
23+
endif::[]
24+
25+
=== Set cluster properties
26+
1827
To connect to a REST catalog, set the following cluster configuration properties:
1928

2029
* config_ref:iceberg_catalog_type,true,properties/cluster-properties[`iceberg_catalog_type`]: `rest`

0 commit comments

Comments
 (0)