Skip to content

doc: nessie doc update #251

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Nov 12, 2025
Merged

doc: nessie doc update #251

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e88f9b6
chore: add global env variable support
schitizsharma Nov 10, 2025
4ebc562
chore: update nessie doc
schitizsharma Nov 10, 2025
9e2195a
Merge remote-tracking branch 'origin' into doc/nessie-doc-update
schitizsharma Nov 10, 2025
b12cfa7
chore: fix nessie version
schitizsharma Nov 10, 2025
ac3038a
chore: remove private keyword
schitizsharma Nov 10, 2025
69a03c1
chore: add bucket name example for local testing
schitizsharma Nov 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 34 additions & 22 deletions docs/writers/iceberg/catalog/rest.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -497,15 +497,16 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al

**Nessie Configuration Parameters:**

| Parameter | Sample Value | Description |
|------------------|-----------------------------------------|-------------------------------------------------------------------------------------------------|
| **REST Catalog URI** | `http://<REST_ENDPOINT>:19120/iceberg/` | Specifies the endpoint URL for the Nessie REST catalog service that the writer will connect to. |
| **Iceberg S3 Path** | `s3://<BUCKET_NAME>` | Determines the S3 path or storage location for Iceberg data in the warehouse bucket. |
| **Iceberg Database** | `iceberg_db` | Specifies the name of the Iceberg database that will be used by the destination configuration. |
| **S3 Endpoint** | `http://<S3_ENDPOINT>:9000` | Endpoint for the S3-compatible service. |
| **AWS Region** | `<S3_REGION>` | Specifies the AWS region associated with the S3 bucket where the data is stored. |
| **AWS Access Key** | `<S3_ACCESS_KEY>` | S3 access key for authentication (Optional). |
| **AWS Secret Key** | `<S3_SECRET_KEY>` | S3 secret key for authentication (Optional). |
| Parameter | Sample Value | Description |
|----------------------|-----------------------------------------|----------------------------------------------------------------------------------------------------------------|
| **catalog_type** | `rest` | Defines the catalog type used by the writer. "rest" means the writer interacts with a RESTful catalog service. |
| **rest_catalog_url** | `http://<YOUR_MACHINE_IP>:19120/iceberg/` | Specifies the endpoint URL for the Nessie REST catalog service. <br/>**Linux:** Use machine's IP (e.g., `http://192.168.1.100:19120/iceberg/`) <br/>**macOS:** Use `http://host.docker.internal:19120/iceberg/` |
| **iceberg_s3_path** | `s3://<BUCKET_NAME>` | Determines the S3 path or storage location for Iceberg data in the warehouse bucket. <br/>Example: `s3://warehouse` for the local testing |
| **iceberg_db** | `<DATABASE_NAME>` | Specifies the name of the Iceberg database that will be used by the destination configuration. <br/>Example: `my_database` |
| **s3_endpoint** | `http://<YOUR_MACHINE_IP>:9000/` | Endpoint for the MinIO S3-compatible service. <br/>**Linux:** Use machine's IP (e.g., `http://192.168.1.100:9000/`) <br/>**macOS:** Use `http://host.docker.internal:9000/` |
| **aws_access_key** | `<S3_ACCESS_KEY>` | MinIO access key for authentication (Optional). <br/>Default from docker-compose: `minio` |
| **aws_secret_key** | `<S3_SECRET_KEY>` | MinIO secret key for authentication (Optional). <br/>Default from docker-compose: `minio123` |
| **aws_region** | `<S3_REGION>` | Specifies the AWS region associated with the S3 bucket where the data is stored. <br/>Example: `us-east-1` |

<br/>

Expand Down Expand Up @@ -537,10 +538,10 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al
"type": "ICEBERG",
"writer": {
"catalog_type": "rest",
"rest_catalog_url": "http://<REST_ENDPOINT>:19120/iceberg/",
"rest_catalog_url": "http://<YOUR_MACHINE_IP>:19120/iceberg/",
"iceberg_s3_path": "s3://<BUCKET_NAME>",
"iceberg_db": "<DATABASE_NAME>",
"s3_endpoint": "http://<S3_ENDPOINT>:9000/",
"s3_endpoint": "http://<YOUR_MACHINE_IP>:9000/",
"aws_access_key": "<S3_ACCESS_KEY>",
"aws_secret_key": "<S3_SECRET_KEY>",
"aws_region": "<AWS_REGION>"
Expand All @@ -549,16 +550,16 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al
```

**Nessie Configuration Parameters**
| Parameter | Sample Value | Description |
|------------------|-----------------------------------------|----------------------------------------------------------------------------------------------------------------|
| Parameter | Sample Value | Description |
|----------------------|-----------------------------------------|----------------------------------------------------------------------------------------------------------------|
| **catalog_type** | `rest` | Defines the catalog type used by the writer. "rest" means the writer interacts with a RESTful catalog service. |
| **rest_catalog_url** | `http://<REST_ENDPOINT>:19120/iceberg/` | Specifies the endpoint URL for the Nessie REST catalog service that the writer will connect to. |
| **iceberg_s3_path** | `s3://<BUCKET_NAME>` | Determines the S3 path or storage location for Iceberg data in the warehouse bucket. |
| **iceberg_db** | `<DATABASE_NAME>` | Specifies the name of the Iceberg database that will be used by the destination configuration. |
| **s3_endpoint** | `http://<S3_ENDPOINT>:9000/` | Endpoint for the MinIO S3-compatible service. |
| **aws_access_key** | `<S3_ACCESS_KEY>` | MinIO access key for authentication (Optional). |
| **aws_secret_key** | `<S3_SECRET_KEY>` | MinIO secret key for authentication (Optional). |
| **aws_region** | `<S3_REGION>` | Specifies the AWS region associated with the S3 bucket where the data is stored. |
| **rest_catalog_url** | `http://<YOUR_MACHINE_IP>:19120/iceberg/` | Specifies the endpoint URL for the Nessie REST catalog service. <br/>**Linux:** Use machine's IP (e.g., `http://192.168.1.100:19120/iceberg/`) <br/>**macOS:** Use `http://host.docker.internal:19120/iceberg/` |
| **iceberg_s3_path** | `s3://<BUCKET_NAME>` | Determines the S3 path or storage location for Iceberg data in the warehouse bucket. <br/>Example: `s3://warehouse` for the local testing |
| **iceberg_db** | `<DATABASE_NAME>` | Specifies the name of the Iceberg database that will be used by the destination configuration. <br/>Example: `my_database` |
| **s3_endpoint** | `http://<YOUR_MACHINE_IP>:9000/` | Endpoint for the MinIO S3-compatible service. <br/>**Linux:** Use machine's IP (e.g., `http://192.168.1.100:9000/`) <br/>**macOS:** Use `http://host.docker.internal:9000/` |
| **aws_access_key** | `<S3_ACCESS_KEY>` | MinIO access key for authentication (Optional). <br/>Default from docker-compose: `minio` |
| **aws_secret_key** | `<S3_SECRET_KEY>` | MinIO secret key for authentication (Optional). <br/>Default from docker-compose: `minio123` |
| **aws_region** | `<S3_REGION>` | Specifies the AWS region associated with the S3 bucket where the data is stored. <br/>Example: `us-east-1` |

<br />

Expand Down Expand Up @@ -591,10 +592,21 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al
2. **PostgreSQL** – Stores Nessie's metadata and catalog state.
3. **MinIO + MinIO Client** – An S3-compatible object store used to store table data and snapshots.

:::note
Before starting the services, update the `nessie.catalog.service.s3.default-options.external-endpoint` configuration in the docker-compose file:

**For Linux machines:**
- Replace `<YOUR_MACHINE_IP>` with machine's private IP address
- Find private IP using: `hostname -I | awk '{print $1}'` or `ip addr show`

**For macOS machines:**
- Replace `<YOUR_MACHINE_IP>` with `host.docker.internal`
:::

```yml title="docker-compose.yml"
services:
nessie:
image: ghcr.io/projectnessie/nessie:latest
image: ghcr.io/projectnessie/nessie:0.105.0
container_name: nessie
ports:
- "19120:19120"
Expand All @@ -609,7 +621,7 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al
- nessie.catalog.service.s3.default-options.region=us-east-1
- nessie.catalog.service.s3.default-options.path-style-access=true
- nessie.catalog.service.s3.default-options.endpoint=http://minio:9000/
- nessie.catalog.service.s3.default-options.external-endpoint=http://localhost:9000/
- nessie.catalog.service.s3.default-options.external-endpoint=http://<YOUR_MACHINE_IP>:9000/
- nessie.catalog.service.s3.default-options.access-key=urn:nessie-secret:quarkus:nessie.catalog.secrets.access-key
- nessie.catalog.secrets.access-key.name=minio
- nessie.catalog.secrets.access-key.secret=minio123
Expand Down