Kafka v2.4.0 introduced MirrorMaker 2.0 (MM2); a new multi-cluster, cross-datacenter replication engine. MirrorMaker 2.0 is well documented in the KIP-382 proposal and the MirrorMaker 2.0 README.md
, but I learn-by-doing.
This demo sets up 2 local, single-broker Kafka clusters and one MirrorMaker 2.0 instance to replicate topics from cluster A to cluster B. Once you have this environment set up, it's quick and easy to play with different configuration options.
Clusters | Topic | Kafka Port | ZooKeeper Port |
---|---|---|---|
A | t1 | 9091 | 2181 |
B | t2 | 9092 | 2182 |
-
Clone this repository and
cd
into the directory. -
Download a Kafka release and extract it to a
./kafka/
. -
Clean up, if you've run this before.
rm -rf /tmp/zk1 rm -rf /tmp/zk2 rm -rf /tmp/kafka-logs-1 rm -rf /tmp/kafka-logs-2
-
Run each of the following commands in a separate shell to start your two single-node Kafka clusters (each with their own single-node ZooKeeper instance).
kafka/bin/zookeeper-server-start.sh zk1.properties kafka/bin/kafka-server-start.sh server1.properties kafka/bin/zookeeper-server-start.sh zk2.properties kafka/bin/kafka-server-start.sh server2.properties
-
Create topics.
kafka/bin/kafka-topics.sh --create --topic t1 --bootstrap-server localhost:9091 --partitions 1 --replication-factor 1 kafka/bin/kafka-topics.sh --create --topic t2 --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
-
Start MirrorMaker.
kafka/bin/connect-mirror-maker.sh mm2.properties
# List topics
kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9091
kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092
# Produce
kafka/bin/kafka-console-producer.sh --topic t1 --bootstrap-server localhost:9091
kafka/bin/kafka-console-producer.sh --topic t2 --bootstrap-server localhost:9092
# Consume
kafka/bin/kafka-console-consumer.sh --topic t1 --bootstrap-server localhost:9091
kafka/bin/kafka-console-consumer.sh --topic t2 --bootstrap-server localhost:9092
# Consumer replicated topic
kafka/bin/kafka-console-consumer.sh --topic A.t1 --bootstrap-server localhost:9092