Apache Kafka + Websockets to Stream Data

Project on implementing streaming of data with Kafka and Websockets in Node JS

YouTube

Set up the demo

1. Install requirements: (node, npm, docker-compose)

2. Install all node projects. From a terminal in the project root folder:

   cd ./backend && npm i
   cd ../frontend && npm i

3. Edit ./docker-compose.yml:20

   • substitute local host IPv4 address

4. Start up the Kafka container

   cd .. && docker-compose up

5. In a new terminal, start up the backend

   # Backend server
   cd ./backend && npm run dev

6. In a new terminal, start up the frontend

   # Front end app
   cd ./frontend && npm run dev

7. Open a browser on http://127.0.0.1:3000/

8. In a new terminal, start up the producer

   # Kafka producer app
   cd ./backend && node src/producer.js

9. Arrange your windows so that browser and producer terminal are both visible.

10. In the producer window, enter data points. Each line contains three space-separated values, followed by a newline. The three values are:

    • [x-value (alpha)] [y-value (number)] [partition id [0,1]]

    • For example:

    a 50 0
    a 50 1
    b 45 0
    b 45 1
    c 55 0
    c 55 1
    

11. To close down the Kafka container, open a terminal on the root folder of the project:

    docker-compose down