This repo provides a demo of the Apache Iceberg Sink Connector for Kafka Connect you can execute locally with docker compose
It extends the docker-spark-iceberg example, so you can also use this environment to test anything related to Iceberg
- redpanda: Just like Kafka but simpler to setup
- connect: Adds the Kafka Connect and the Iceberg Sink Connector settings
- console: Redpanda UI
- minio: Object storage compatible with Amazon S3
- aws: AWS CLI to create a Minio bucket
- spark-iceberg: Spark + Iceberg environment
- rest: Rest catalog to interact with Iceberg
You can acces the Redpanda UI at http://localhost:18080, there's two main screens you should look at, Topics and Connectors
The control-iceberg topic it's used by the connector while payments is the one you going to publish the messages
You'll probabily want to wait until the IcebergSinkConnector be up and running before start publishing at the Topic, just like the image below
During the producer execution you can also monitor the screen to check if the connector got any error
You can acces the Minio UI at http://localhost:9000, it will require an user and a password, both are set to minioadmin
The AWS CLI will create a bucket called warehouse during the docker compose execution, also a dataset and a table will be created by the spark-iceberg container
The script that creates both dataset and table can be found at /spark/create_table.py
You can open the jupyter lab at http://localhost:8888/lab/tree/notebooks and use the sample notebook to query the table
Refer to docker-spark-iceberg to check more details about it
Benthos it's declarative way of building stream processing and is used as the message producer in this example
The benthos.json.kafka.yaml file describes the pipeline, it will consume from the file located at data/* and will publish to the payments Kafka topic
Installing:
curl -Lsf https://sh.benthos.dev | bash
Running:
benthos -c ./benthos.json.kafka.yaml
To make it simpler, you can also run with make produce
