You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+59-54
Original file line number
Diff line number
Diff line change
@@ -5,93 +5,74 @@ This example has two components to demonstrate the utilities.
5
5
6
6
Debezium
7
7
8
-
Requirements
9
-
Just docker
8
+
Requirements:<br>
9
+
-Just docker
10
10
11
-
12
-
Create your Django app:
13
-
14
-
docker-compose run --rm --no-deps web django-admin startproject django_cdc .
15
-
16
-
Modify src/django_cdc/settings.py
17
-
18
-
DATABASES = {
19
-
'default': {
20
-
'ENGINE': 'django.db.backends.mysql',
21
-
'NAME': 'djangodb',
22
-
'USER': 'django',
23
-
'PASSWORD': 'django',
24
-
'HOST': 'mysql',
25
-
'PORT': 3306,
26
-
}
27
-
}
28
-
29
-
Run:
11
+
To get the docker containers up and running:
30
12
31
13
docker-compose up -d
32
14
33
-
Run for default django tables:
15
+
To create the django tables in MySQL:
34
16
35
17
docker-compose run --rm --no-deps python_app python manage.py migrate
36
-
docker-compose run --rm --no-deps python_app python manage.py makemigrations polls
37
-
docker-compose run --rm --no-deps python_app python manage.py migrate polls
38
18
39
-
Add some polls from admin page
19
+
To add some polls from admin page, create a superuser
40
20
41
21
docker-compose run --rm --no-deps python_app python manage.py createsuperuser
42
22
43
-
The default login and password for the admin site is admin:admin.
23
+
Using the username/password you just generated, you can later visit http://localhost:8000/admin/polls/question/ and create some rows after setting up CDC.
24
+
25
+
Grant the required MySQL rights to django so that Debezium can do it's job.
26
+
To do this, go to Adminer UI at http://localhost:8080/. Login using:
44
27
45
-
Grant the required MySQL rights to django so that CDC can do it's job:
28
+
Server: mysql
29
+
Username: root
30
+
Password: pass
31
+
Database: djangodb
32
+
33
+
After logging in, click "SQL command" and execute this:
46
34
47
35
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO django@'%';
48
36
49
-
Verify Debezium MySQL connector configuration. Read more about it at https://debezium.io/documentation/reference/connectors/mysql.html#mysql-required-connector-configuration-properties
37
+
The next thing to do is set up Debezium by sending a cURL command to kafka connect.<br>
38
+
You can read about the Debezium MySQL connector configuration at https://debezium.io/documentation/reference/connectors/mysql.html#mysql-required-connector-configuration-properties
39
+
40
+
To send our binary Protobuffer data, we will use the same method as Avro configuration explained here: https://debezium.io/documentation/reference/transformations/outbox-event-router.html#avro-as-payload-format
50
41
51
-
{
52
-
"name": "cdc-python-netcore-connector",
42
+
Open a new terminal, and use the curl command to register the Debezium MySQL connector. (You may need to escape your double-quotes on windows if you get a parsing error). This will add a connector in our kafka-connect container to listen to database changes in our outbox table.
Open a new terminal, and use the curl command to register the Debezium MySQL connector. (You may need to escape your double-quotes on windows if you get a parsing error)
I created a protobuf file to have a well-defined type between python and .net core: `/proto/question.proto`. To compile the proto file, you can install the protobuf compiler using:
69
+
To prepare a protobuf file between python and .net core, I wrote a proto file: `/proto/question.proto`. To compile the proto file, you can install the protobuf compiler using:
89
70
90
71
brew install protobuf
91
72
92
73
And run the following command inside the proto folder (I've already included the compiled output of the proto file in the repo).
We now need an outbox table to implement the cdc outbox pattern using Debezium. I created the Outbox model for this:
97
78
@@ -103,3 +84,27 @@ We now need an outbox table to implement the cdc outbox pattern using Debezium.
103
84
payload = models.BinaryField()
104
85
105
86
You can read the [Debezium documentation](https://debezium.io/documentation/reference/configuration/outbox-event-router.html) for details. The `payload` column is special here since it will hold the serialized protobuf value and it will be passed transparently by Debezium to Kafka.
87
+
88
+
To populate the outbox table, I used the `save_model` method of admin view:
0 commit comments