Sample-Kafka-Application

Instructions along with sample code base for basic kafka project

#Source: https://www.udemy.com/apache-kafka/

Installation Steps in Mac OS X

• Install brew if needed: /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
• Download and Setup Java 8 JDK:
  1. brew tap caskroom/versions
  2. brew cask install java8
• Download & Extract the Kafka binaries from https://kafka.apache.org/downloads
• Install Kafka commands using brew: brew install kafka
• Create data/zookeeper, data/kafka in folder
• Edit Zookeeper & Kafka configs using a text editor
  1. zookeeper.properties: dataDir=/your/path/to/data/zookeeper
  2. server.properties: log.dirs=/your/path/to/data/kafka

Sample Command Lines

• Start Zookeeper (run in installed kafka directory): zookeeper-server-start config/zookeeper.properties
• Start Kafka (run in installed kafka directory): kafka-server-start config/server.properties
• Kafka Topics
  1. Create Kafka Topic: kafka-topics --zookeeper 127.0.0.1:2181 --topic topic1 --create --partitions 3 --replication-factor 1
  2. List Kafka Topics: kafka-topics --zookeeper 127.0.0.1:2181 --list
  3. Describe Topic: kafka-topics --zookeeper 127.0.0.1:2181 --topic topicName --describe
  4. Delete Topic: kafka-topics --zookeeper 127.0.0.1:2181 --topic topicName --delete
• Kafka Producer Consumer
  1. produce message for a particular topic: kafka-console-producer --broker-list 127.0.0.1:9092 --topic topic1
  2. Consume message from topic from beginning: kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic topic1 --from-beginning
  3. Consume with consumer group: kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic topic1 --group consumer-group-1
  4. List all consumer groups: kafka-consumer-groups --bootstrap-server localhost:9092 --list