Skip to content

Commit

Permalink
major tutorial upgrade to DBZ 2.6.0.Final (#1)
Browse files Browse the repository at this point in the history
* dbz 2.6 upgrade and zookeeper removal for dbz ui example

* dbz 2.6 upgrade and zookeeper removal for kafka connect example

* dbz 2.6 upgrade and zookeeper removal for dbz server example

* update all images from 2.6 to 2.6.0.Final after availability on quay

* small fixes for dbz-ui example doc
  • Loading branch information
hpgrahsl authored Apr 4, 2024
1 parent 4073cee commit 84aa211
Show file tree
Hide file tree
Showing 10 changed files with 370 additions and 269 deletions.
28 changes: 28 additions & 0 deletions apps/infra/elementary/dbz-server/config/application.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Quarkus settings

quarkus.log.console.json=false

# Debezium Server settings
debezium.format.key=json
debezium.format.value=json

# Debezium Server source settings for postgreSQL

debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector
debezium.source.offset.storage.file.filename=/debezium/data/offsets.dat
debezium.source.offset.flush.interval.ms=0
debezium.source.database.hostname=postgres
debezium.source.database.port=5432
debezium.source.database.user=postgres
debezium.source.database.password=postgres
debezium.source.database.dbname=postgres
debezium.source.topic.prefix=dbzserver
debezium.source.schema.include.list=inventory
debezium.source.table.include.list=inventory.customers
debezium.source.slot.name=dbzserver123

# Debezium Server sink settings for an HTTP endpoint/service
# use your individual http url show in your webhook.site browser tab

debezium.sink.type=http
debezium.sink.http.url=https://webhook.site/<YOUR_INDIVIDUAL_UUID_HERE>
Empty file.
23 changes: 11 additions & 12 deletions apps/infra/elementary/dbz-ui/compose.yaml
Original file line number Diff line number Diff line change
@@ -1,24 +1,22 @@
version: '3'
services:
zookeeper:
container_name: zookeeper
image: quay.io/debezium/zookeeper:2.2.0.Final
networks:
- demo-network
kafka:
container_name: kafka
image: quay.io/debezium/kafka:2.2.0.Final
image: quay.io/strimzi/kafka:0.40.0-kafka-3.7.0
command:
- /bin/sh
- -c
- './bin/kafka-storage.sh format -t 9EDB9GBlQASth3IVRW0BCw -c ./config/kraft/server.properties && ./bin/kafka-server-start.sh ./config/kraft/server.properties --override advertised.listeners=$${KAFKA_ADVERTISED_LISTENERS}'
ports:
- "9092:9092"
depends_on:
- zookeeper
environment:
- ZOOKEEPER_CONNECT=zookeeper:2181
- LOG_DIR=/tmp/logs
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
networks:
- demo-network
- demo-network
mysql:
container_name: mysql
image: quay.io/debezium/example-mysql:2.2.0.Final
image: quay.io/debezium/example-mysql:2.6.0.Final
ports:
- "3306:3306"
environment:
Expand All @@ -29,7 +27,7 @@ services:
- demo-network
connect:
container_name: connect
image: quay.io/debezium/connect:2.2.0.Final
image: quay.io/debezium/connect:2.6.0.Final
ports:
- "8083:8083"
depends_on:
Expand All @@ -48,6 +46,7 @@ services:
- demo-network
debezium-ui:
container_name: debezium-ui
#NOTE: tag '2.5' is the most recent currently but has no arm image on quay yet hence sticking to 2.2
image: quay.io/debezium/debezium-ui:2.2
ports:
- "8080:8080"
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
179 changes: 87 additions & 92 deletions documentation/modules/ROOT/pages/20-dbz-kafka-connect.adoc

Large diffs are not rendered by default.

338 changes: 208 additions & 130 deletions documentation/modules/ROOT/pages/21-dbz-server.adoc

Large diffs are not rendered by default.

71 changes: 36 additions & 35 deletions documentation/modules/ROOT/pages/22-dbz-ui.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@ NOTE: Make sure that besides Podman (Desktop) you also have `podman-compose` ins
[.console-input]
[source,adoc]
----
podman-compose -f apps/infra/elementary/dbz-ui/compose.yaml up
podman compose -f apps/infra/elementary/dbz-ui/compose.yaml up
----
Quickly check if all 5 containers are up and running fine.
Quickly check if all 4 containers are up and running fine.
[.console-input]
[source,adoc]
Expand All @@ -33,12 +33,11 @@ podman ps
[.console-output]
[source,adoc]
----
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1ec62b46c48a quay.io/debezium/zookeeper:2.2 start 6 days ago Up 6 days zookeeper
a1d433e088df quay.io/debezium/example-mysql:2.2 mysqld 6 days ago Up 6 days 0.0.0.0:3306->3306/tcp mysql
3af157e33520 quay.io/debezium/kafka:2.2 start 6 days ago Up 6 days 0.0.0.0:9092->9092/tcp kafka
18fa09d1b116 quay.io/debezium/connect:2.2 start 6 days ago Up 6 days 0.0.0.0:8083->8083/tcp connect
a59d2286e7d1 quay.io/debezium/debezium-ui:2.2 6 days ago Up 6 days 0.0.0.0:8080->8080/tcp debezium-ui
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c5bcbd907ab7 quay.io/strimzi/kafka:0.40.0-kafka-3.7.0 /bin/sh -c ./bin/... 15 seconds ago Up 11 seconds 0.0.0.0:9092->9092/tcp kafka
85198a7a8c97 quay.io/debezium/example-mysql:2.6.0.Final mysqld 14 seconds ago Up 10 seconds 0.0.0.0:3306->3306/tcp mysql
808a633208b7 quay.io/debezium/connect:2.6.0.Final start 12 seconds ago Up 9 seconds 0.0.0.0:8083->8083/tcp connect
4a401630ea8d quay.io/debezium/debezium-ui:2.2 11 seconds ago Up 8 seconds 0.0.0.0:8080->8080/tcp debezium-ui
----
--
Expand All @@ -51,7 +50,7 @@ Docker::
docker compose -f apps/infra/elementary/dbz-ui/compose.yaml up
----
Quickly check if all 5 containers are up and running fine.
Quickly check if all 4 containers are up and running fine.
[.console-input]
[source,adoc]
Expand All @@ -62,12 +61,11 @@ docker ps
[.console-output]
[source,adoc]
----
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
45e4b8b1cd3a quay.io/debezium/debezium-ui:2.2 "/deployments/run-ja…" 2 seconds ago Up Less than a second 0.0.0.0:8080->8080/tcp debezium-ui
4ff3b282bd2b quay.io/debezium/connect:2.2 "/docker-entrypoint.…" 3 seconds ago Up 1 second 0.0.0.0:8083->8083/tcp, 9092/tcp connect
8150278a7fde quay.io/debezium/kafka:2.2 "/docker-entrypoint.…" 3 seconds ago Up 1 second 0.0.0.0:9092->9092/tcp kafka
c50d16e368c5 quay.io/debezium/example-mysql:2.2 "docker-entrypoint.s…" 3 seconds ago Up 1 second 0.0.0.0:3306->3306/tcp, 33060/tcp mysql
e2a09a877e16 quay.io/debezium/zookeeper:2.2 "/docker-entrypoint.…" 3 seconds ago Up 1 second 2181/tcp, 2888/tcp, 3888/tcp zookeeper
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c5bcbd907ab7 quay.io/strimzi/kafka:0.40.0-kafka-3.7.0 /bin/sh -c ./bin/... 15 seconds ago Up 11 seconds 0.0.0.0:9092->9092/tcp kafka
85198a7a8c97 quay.io/debezium/example-mysql:2.6.0.Final mysqld 14 seconds ago Up 10 seconds 0.0.0.0:3306->3306/tcp mysql
808a633208b7 quay.io/debezium/connect:2.6.0.Final start 12 seconds ago Up 9 seconds 0.0.0.0:8083->8083/tcp connect
4a401630ea8d quay.io/debezium/debezium-ui:2.2 11 seconds ago Up 8 seconds 0.0.0.0:8080->8080/tcp debezium-ui
----
--
Expand Down Expand Up @@ -194,13 +192,15 @@ NOTE: The schema information in the payload has been removed from the JSON docum
"email": "annek@noanswer.org"
},
"source": {
"version": "2.2.0.Final",
"version": "2.6.0.Final",
"connector": "mysql",
"name": "mydb1",
"ts_ms": 1690555772000,
"ts_ms": 1712130748000,
"snapshot": "last_in_data_collection",
"db": "inventory",
"sequence": null,
"ts_us": 1712130748000000,
"ts_ns": 1712130748000000000,
"table": "customers",
"server_id": 0,
"gtid": null,
Expand All @@ -211,7 +211,9 @@ NOTE: The schema information in the payload has been removed from the JSON docum
"query": null
},
"op": "r",
"ts_ms": 1690555772308,
"ts_ms": 1712130748764,
"ts_us": 1712130748764950,
"ts_ns": 1712130748764950000,
"transaction": null
}
}
Expand Down Expand Up @@ -264,13 +266,15 @@ NOTE: The schema information in the payload has been removed from the JSON docum
"type": "LIVING"
},
"source": {
"version": "2.2.0.Final",
"version": "2.6.0.Final",
"connector": "mysql",
"name": "mydb1",
"ts_ms": 1690555772000,
"ts_ms": 1712130748000,
"snapshot": "last",
"db": "inventory",
"sequence": null,
"ts_us": 1712130748000000,
"ts_ns": 1712130748000000000,
"table": "addresses",
"server_id": 0,
"gtid": null,
Expand All @@ -281,7 +285,9 @@ NOTE: The schema information in the payload has been removed from the JSON docum
"query": null
},
"op": "r",
"ts_ms": 1690555772330,
"ts_ms": 1712130748768,
"ts_us": 1712130748768202,
"ts_ns": 1712130748768202000,
"transaction": null
}
}
Expand All @@ -301,45 +307,41 @@ NOTE: Make sure that besides Podman (Desktop) you also have `podman-compose` ins
[.console-input]
[source,adoc]
----
podman-compose -f apps/infra/elementary/dbz-ui/compose.yaml down
podman compose -f apps/infra/elementary/dbz-ui/compose.yaml down
----
[.console-output]
[source,adoc]
----
>>>> Executing external compose provider "/opt/homebrew/bin/podman-compose". Please refer to the documentation for details. <<<<
podman-compose version: 1.0.6
['podman', '--version', '']
using podman version: 4.5.1
using podman version: 5.0.1
** excluding: set()
podman stop -t 10 debezium-ui
debezium-ui
exit code: 0
podman stop -t 10 connect
connect
exit code: 0
podman stop -t 10 kafka
kafka
exit code: 0
podman stop -t 10 mysql
mysql
exit code: 0
podman stop -t 10 zookeeper
zookeeper
podman stop -t 10 kafka
kafka
exit code: 0
podman rm debezium-ui
debezium-ui
exit code: 0
podman rm connect
connect
exit code: 0
podman rm kafka
kafka
exit code: 0
podman rm mysql
mysql
exit code: 0
podman rm zookeeper
zookeeper
podman rm kafka
kafka
exit code: 0
----
Expand All @@ -355,12 +357,11 @@ docker compose -f apps/infra/elementary/dbz-ui/compose.yaml down
[.console-output]
[source,adoc]
----
[+] Running 6/6
[+] Running 5/5
✔ Container debezium-ui Removed 0.2s
✔ Container connect Removed 0.6s
✔ Container mysql Removed 1.0s
✔ Container kafka Removed 1.1s
✔ Container zookeeper Removed 0.5s
✔ Network dbz-ui Removed
----
--
Expand Down

0 comments on commit 84aa211

Please sign in to comment.