Skip to content

Kafka

Jump to bottom
mrb-github edited this page Apr 24, 2020 · 19 revisions

Goal: Explore Kafka, run performance test, document results.

Excerpts: I have 15 years of experience with multiple messaging systems, starting with Tibco, Tibco JMS, SonicMQ, WebsphereMQ, Informatica (LBM/29West) UME, Ionic. I have recently explored the AmazonMQ service. All except Informatica UME are TCP based systems with brokers, whereas UME is multicast. The highest performance was exhibited by UME because it is brokerless and the message duplication happens at the switch layer. We routinely exercised and used the system with 30+ sources (receivers) and 200+ receivers (senders). Thus, it is interesting to see how the Kafka performs.

Steps:

  1. Downloaded the zip file from: https://www.apache.org/info/verification.html
  2. Started a separate Ec2 instance where I plan to host the Kafka broker
  3. unzip the file. This unzips in home dir: /home/ubuntu command: tar -xzf kafka_2.12-2.5.0.tgz
  4. Copy the directory to /usr/bin and make it executable and add it to PATH
  5. Start zookeeper: zookeeper-server-start.sh -daemon /usr/bin/kafka_2.12-2.5.0/config/zookeeper.properties
  6. Edit the kafka-server-start.sh set start and Max memory to be used by JVM to 512MB instead of 1G as my free Ec2 Instance didn't have that much memory export KAFKA_HEAP_OPTS="-Xmx512M -Xms512M" #export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
  7. Start kafka: kafka-server-start.sh -daemon /usr/bin/kafka_2.12-2.5.0/config/server.properties
  8. kafka port: 9902

Sanity Check:

  1. kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
  2. kafka-console-producer.sh --topic test --broker-li"localhost:9092"

Problem:1 ubuntu@ip-172-31-21-208:~$ gpg --import KEYS gpg: can't open 'KEYS': No such file or directory gpg: Total number processed: 0

Solution: Resolved to checking the hash on the file ubuntu@ip-172-31-21-208:~$ sha256sum kafka_2.12-2.5.0.tgz 4a857*******04baf kafka_2.12-2.5.0.tgz ubuntu@ip-172-31-21-208: gpg --print-md SHA256 kafka_2.12-2.5.0.tgz kafka_2.12-2.5.0.tgz: 4A857XXXXX4BAF

Problem:2 sh /usr/bin/kafka_2.12-2.5.0/config/zookeeper.properties /usr/bin/kafka_2.12-2.5.0/bin/kafka-run-class.sh: line 315: exec: java: not found Solution: Install Java sudo apt-get update --fix-missing sudo apt install default-jre
java -version

Clone this wiki locally