Skip to content

Kafka cluster

Jump to bottom
stanislawbartkowski edited this page Feb 16, 2022 · 25 revisions

Introduction

Short description on how to set up minimalistic Kafka cluster

Zookeeper

Zookeeper hosts

At least three nodes to install HA Zookeeper cluster.

cat /etc/hosts

192.168.0.13	zoo1.sb.com	zoo1
192.168.0.76	zoo2.sb.com	zoo2
192.168.0.111	zoo3.sb.com	zoo3

Download and install

https://zookeeper.apache.org/releases.html#download

Unpack in /opt directory

ls /opt/apache-zookeeper-3.7.0-bin/ -ltr

* bin
* README_packaging.md
* README.md
* NOTICE.txt
* LICENSE.txt
* docs
* lib
* conf
* logs

Starter

Create shortcut in /usr/local/bin

cat zkServer

cd /opt/apache-zookeeper-3.7.0-bin/bin
exec ./zkServer.sh $@

Configure cluster

On every host in the cluster create the corresponding myid tag file as "1", "2" and "3". Directory for myid file is defined by dataDir configuration parameter in zoo.cfg file.

mkdir /var/lib/zookeeper
echo "1" > /var/lib/zookeeper/myid

cd /opt/apache-zookeeper-3.7.0-bin/conf
cp cp zoo_sample.cfg zoo.cfg
vi zoo.cfg

dataDir=/var/lib/zookeeper

server.1=zoo1.sb.com:2888:3888
server.2=zoo2.sb.com:2888:3888
server.3=zoo3.sb.com:2888:3888

Start

On all three nodes run.

zkServer start

After several seconds verify.

zkServer status

On server should report leader mode and two other follower modes.

zkServer status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/apache-zookeeper-3.7.0-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader

Kafka cluster

Kafka hosts

At least three hosts.

cat /etc/hosts


192.168.122.73	kafka1.sb.com	kafka1
192.168.122.246	kafka2.sb.com	kafka2
192.168.122.176	kafka3.sb.com	kafka3

Download and install

https://kafka.apache.org/downloads

Untar in /opt directory

ls /opt/kafka_2.13-3.0.0/ 

* NOTICE
* LICENSE
* licenses
* configvi server.properties
* bin
* site-docs
* libs

Create starter

cd /usr/local/bin
cat kafkaStart

cd /opt/kafka_2.13-3.0.0/
exec bin/kafka-server-start.sh config/server.properties

Configure

cd /opt/kafka_2.13-3.0.0/config

Parameter zookeeper.connect specifies Zookeeper cluster. Parameter broker.id should be different for every host in the cluster. For instance: 0,1 and 2.

vi server.properties

zookeeper.connect=zoo1.sb.com:2181,zoo2.sb.com:2181,zoo3.sb.com:2181
.........
broker.id=0

Start

kafkaStart

Basic test

Create topic

kafka-topics --create --partitions 1 --replication-factor 1 --topic quickstart-events --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092

Created topic quickstart-events.

List topics.

kafka-topics --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --list

quickstart-events

Publish messages to topic

kafka-console-producer --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --topic quickstart-events

>first message
>second message
>third message

Publish messages as key/value

kafka-console-producer --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --topic test -property parse.key=true --property key.separator=,

key1,line1
key2,line2
key3,line3

Read messages for topic

kafka-console-consumer --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --topic quickstart-events --from-beginning

first message
second message
third message

Read messages including key

kafka-console-consumer --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --topic test --from-beginning --property print.key=true

Useful commands

Detailed information about the topic.

kafka-topics --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --describe --topic quickstart-events

Topic: quickstart-events	TopicId: Ru_ak3wiSseoWMCHs7RVug	PartitionCount: 1	ReplicationFactor: 1	Configs: segment.bytes=1073741824
	Topic: quickstart-events	Partition: 0	Leader: 2	Replicas: 2	Isr: 2

(There is a single replica of the topic on the Kafka broker 2)

Delete the topic

kafka-topics --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --delete --topic quickstart-events

Create a topic with a replication factor

kafka-topics --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --create --partitions 1 --topic quickstart-events --replication-factor 3

kafka-topics --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --describe --topic quickstart-events

Topic: quickstart-events	TopicId: _UMqCSphShWF0NZiQEwndA	PartitionCount: 1	ReplicationFactor: 3	Configs: segment.bytes=1073741824
	Topic: quickstart-events	Partition: 0	Leader: 0	Replicas: 0,2,1	Isr: 0,2,1

(three replicas across three Kafka brokers)

Consumer groups

kafka-consumer-groups.sh --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --describe --all-groups

Offsets

kafka-get-offsets.sh --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --topic test

Clone this wiki locally