|
This repository gathers Kafka code examples coming from various websites or written by ourself. In particular it includes several build scripts (bash scripts, batch files) for experimenting with the Kafka system on a Windows machine. |
Ada, Akka, C++, COBOL, Dafny, Dart, Deno, Docker, Erlang, Flix, Golang, GraalVM, Haskell, Kotlin, LLVM, Modula-2, Node.js, Rust, Scala 3, Spark, Spring, TruffleSqueak, WiX Toolset and Zig are other topics we are continuously monitoring.
☛ Read the document "Getting Started" from the Kafka documentation to know more about the Kafka ecosystem.
Project dependencies ▴
This project depends on the following external software for the Microsoft Windows platform:
- Git 2.47 (release notes)
- Kafka 3.8 1 (release notes)
- Temurin OpenJDK 17 LTS 2 (release notes, bug fixes)
Optionally one may also install the following software:
- Apache Maven 3.9 (requires Java 8+) (release notes)
- ConEmu 2023 (release notes)
- Gradle 8.10 (requires Java 8+) (release notes)
- Offset Explorer 3.0 3 (change history)
- Scala 2.13 (requires Java 8+) (release notes, Scala API)
- Temurin OpenJDK 21 LTS (release notes, Java 21 API)
- Visual Studio Code 1.94 (release notes)
☛ Installation policy
When possible we install software from a Zip archive rather than via a Windows installer. In our case we definedC:\opt\as the installation directory for optional software tools (in reference to the/opt/directory on Unix).
For instance our development environment looks as follows (November 2024) 4:
C:\opt\apache-maven\ ( 10 MB) C:\opt\ConEmu\ ( 26 MB) C:\opt\Git\ (387 MB) C:\opt\gradle\ (135 MB) C:\opt\jdk-temurin-17.0.13_11\ (301 MB) C:\opt\jdk-temurin-21.0.5_11\ (325 MB) C:\opt\kafka_2.13-3.8.1\ (120 MB) C:\opt\scala-2.13.15\ ( 24 MB) C:\opt\VSCode\ (341 MB) C:\Program Files\OffsetExplorer2\ (112 MB)
🔎 Git for Windows provides a BASH emulation used to run
gitfrom the command line (as well as over 250 Unix commands likeawk,diff,file,grep,more,mv,rmdir,sedandwc).
Directory structure ▴
This project has following directory structure :
docs\
examples\{README.md, quickstart, ..}
kafka\ (git module)
README.md
RESOURCES.md
setenv.bat
where
- directory
docs\contains Kafka related papers/articles. - directory
examples\contains Kafka examples grabbed from various websites (see fileexamples\README.md). - file
README.mdis the Markdown document for this page. - file
RESOURCES.mdgathers Kafka related informations. - file
setenv.batis the batch command for setting up our environment.
We also define a virtual drive – e.g. drive K: – in our working environment in order to reduce/hide the real path of our project directory (see article "Windows command prompt limitation" from Microsoft Support).
🔎 We use the Windows external command
substto create virtual drives; for instance:> subst K: %USERPROFILE%\workspace\kafka-examples
In the next section we give a brief description of the batch files present in this project.
Batch commands ▴
We execute command setenv.bat once to setup our development environment; it makes external tools such as git.exe, gradle.bat and sh.exe directly available from the Windows command prompt (see section Project dependencies).
> setenv -verbose Tool versions: java 17.0.13, javac 17.0.13, scalac 2.13.15, gradle 8.10.2, kafka-configs 3.8.0, mvn 3.9.9, git 2.47.0, diff 3.10, bash 5.2.32(1) Tool paths: C:\opt\jdk-temurin-17.0.13_11\bin\java.exe C:\opt\jdk-temurin-17.0.13_11\bin\javac.exe C:\opt\scala-2.13.15\bin\scalac.bat C:\opt\gradle\bin\gradle.bat C:\opt\kafka_2.13-3.8.1\bin\windows\kafka-configs.bat C:\opt\apache-maven\bin\mvn.cmd C:\opt\Git\bin\git.exe C:\opt\Git\usr\bin\diff.exe C:\opt\Git\bin\bash.exe Environment variables: "GIT_HOME=C:\opt\Git" "GRADLE_HOME=C:\opt\gradle" "JAVA_HOME=C:\opt\jdk-temurin-17.0.13_11" "KAFKA_HOME=C:\opt\kafka_2.13-3.8.1" "MAVEN_HOME=C:\opt\apache-maven" "SCALA_HOME=C:\opt\scala-2.13.15" > where git gradle C:\opt\Git\bin\git.exe C:\opt\Git\mingw64\bin\git.exe C:\opt\gradle\bin\gradle C:\opt\gradle\bin\gradle.bat C:\opt\Git\bin\sh.exe C:\opt\Git\usr\bin\sh.exe
Footnotes ▴
[1] Kafka components ↩
-
Kakfa ZooKeeper Jetty Java 3.8.0 3.8.4 9.4.54 b) 8+ 3.7.1 3.8.3 9.4.54 b) 8+ 3.7.0 3.8.4 9.4.54 b) 8+ 3.6.2 3.8.4 a) 8+ 3.6.1 3.8.3 9.4.53 b) 8+ 3.6.0 3.8.2 9.4.52 8+ 3.5.1 3.6.4 9.4.51 8+ 3.5.0 3.6.4 9.4.51 8+ 3.4.1 3.6.4 9.4.51 8+ 3.4.0 3.6.3 9.4.48 8+ 3.3.x 3.6.3 ? ? 3.2.x 3.6.3 ? ?
b) Jetty current version is 12.0.12 as of July 30, 2024 (MVN Repository).
c) Jetty 9.4.x is at End of Community Support as of June 1, 2023 (#7958).
[2] Java compatibility ↩
- KIP-750: Java 8 support is deprecating in Apache Kafka 3.0 and will be dropped in Apache Kafka 4.0.
[3] Offset Explorer ↩
-
Offset Explorer (formerly named Kalfa Tool) provides the following features :
- The browser tree allows us to easily view and navigate the objects (brokers, topices, particitions, consumers) in our Apache Kafka cluster.
- The explorer allows us to view messages (and their keys) in the partitions of the topics we are interested in.
- If the string-based data is either in JSON or XML format, we can view it in a pretty-printed manner.
- The browser tree also allows us to quickly view all offsets of our Kafka consumers.
[4] Downloads ↩
- In our case we downloaded the following installation files (see section 1):
-
apache-maven-3.9.9-bin.zip ( 10 MB) ConEmuPack.230724.7z ( 5 MB) gradle-8.10.2-bin.zip (118 MB) kafka_2.13-3.8.1.tgz ( 82 MB) offsetexplorer_64bit.exe ( 37 MB) OpenJDK11U-jdk_x64_windows_hotspot_11.0.23_9.zip ( 99 MB) OpenJDK17U-jdk_x64_windows_hotspot_17.0.13_11.zip ( 99 MB) OpenJDK21U-jdk_x64_windows_hotspot_21.0.4_7.zip (191 MB) PortableGit-2.47.0-64-bit.7z.exe ( 41 MB) scala-2.13.15.zip ( 22 MB) VSCode-win32-x64-1.91.1.zip (131 MB)