Spring Boot Kafka Microservice with Observability Stack

This project demonstrates a microservice architecture using Spring Boot, Apache Kafka, and an observability stack.

🎯 Objectives

This project demonstrates:

• Microservice Architecture: Service decomposition and communication
• Event-Driven Design: Asynchronous processing with Kafka
• Observability: Monitoring, logging, and distributed tracing
• Containerization: Docker setup
• Configuration Management: Environment-based configuration
• Documentation: API documentation

🏷️ Technologies Used

• Framework: Spring Boot 3.5.5
• Language: Java 17
• Message Broker: Apache Kafka 4.1.0
• Build Tool: Gradle
• Containerization: Docker & Docker Compose
• Monitoring: Prometheus, Grafana, Loki, Promtail
• Distributed Tracing: Zipkin with Micrometer Tracing
• Documentation: OpenAPI 3 / Swagger

🏗️ Services Overview

This project demonstrates an event-driven microservice architecture with the following components:

• notification-gateway: REST API that receives requests and publishes to Kafka
• notification-dispatcher: Kafka consumer that processes notification events
• common-lib: Shared DTOs and models
• Observability Stack: Prometheus, Grafana, Loki, Promtail, and Zipkin

🚀 Quick Start

Prerequisites

• Java 17+
• Docker & Docker Compose
• Gradle (optional, using wrapper)

Note: Includes VS Code devcontainer with development tools.

Running the Application

1. Clone the repository

   git clone <repository-url>
   cd spring-boot-kafka-microservice-example

2. Start all services with Docker Compose

   docker-compose up -d

3. Verify services are running

   docker-compose ps

🎯 Testing the System

Send a notification via REST API:

curl -X POST http://localhost:8080/api/notifications \
  -H "Content-Type: application/json" \
  -d '{
    "type": "EMAIL",
    "to": "user@example.com",
    "message": "Hello from Spring Boot Kafka!"
  }'

Expected Response:

{
  "id": "generated-uuid",
  "message": "Notification queued successfully",
  "timestamp": "2025-09-22T10:30:00Z"
}

The notification will be:

1. Received by the notification-gateway
2. Published to Kafka topic notifications
3. Consumed by notification-dispatcher
4. Processed based on notification type (EMAIL, WEB, PUSH)

🔧 Services Configuration

notification-gateway (Port 8080)

• Purpose: REST API for receiving notification requests
• Tech Stack: Spring Boot, Spring WebFlux, Kafka Producer
• Health Check: http://localhost:8080/actuator/health
• API Documentation: Swagger UI is available at startup (http://localhost:8080/swagger-ui/index.html)

notification-dispatcher (Port 8081)

• Purpose: Kafka consumer for processing notifications
• Tech Stack: Spring Boot, Kafka Consumer, Strategy Pattern
• Health Check: http://localhost:8081/actuator/health
• Processing: Supports EMAIL, WEB, and PUSH notification types

Kafka (Port 9092)

• Broker: Apache Kafka 4.1.0
• Topic: notifications (auto-created)
• UI: Kafdrop is available at http://localhost:9000

📊 Observability & Monitoring

Access the monitoring stack:

Service	URL	Credentials	Purpose
Grafana	http://localhost:3000	admin/admin123	Dashboards & Visualization
Prometheus	http://localhost:9090	-	Metrics Collection
Zipkin	http://localhost:9411	-	Distributed Tracing
Kafdrop	http://localhost:9000	-	Kafka Topic Browser

🔍 What's Being Monitored

• Application metrics (JVM, HTTP requests)
• Kafka metrics (producer/consumer performance)
• System metrics (CPU, memory)
• Logs with trace correlation
• Request tracing across services

📈 Available Grafana Dashboards

The project includes pre-configured Grafana dashboards:

• Notification Service Logs: Service logs with filtering and trace correlation
• API Flow Monitoring: Real-time monitoring of the notification flow
• Consumer Processing: Kafka consumer processing logs and metrics
• Trace ID Search: Search and correlate logs by trace ID across services

Access dashboards at: http://localhost:3000 (admin/admin123)

🛠️ Development

Building the Application

# Build all modules
./gradlew build

# Build specific module
./gradlew :notification-gateway:build
./gradlew :notification-dispatcher:build

Local Development (without Docker)

1. Start Kafka locally

   docker-compose up -d kafka kafdrop

2. Run services with Gradle

   # Terminal 1 - Gateway
   ./gradlew :notification-gateway:bootRun
   
   # Terminal 2 - Dispatcher
   ./gradlew :notification-dispatcher:bootRun

🔧 Configuration

Environment Variables

Variable	Default Value	Description
KAFKA_BOOTSTRAP	kafka:9092	Kafka broker connection string
NOTIFICATIONS_TOPIC	notifications	Kafka topic name
ZIPKIN_ENDPOINT	http://zipkin:9411/api/v2/spans	Zipkin tracing endpoint
TRACING_SAMPLING_PROBABILITY	1.0	Trace sampling rate (0.0 to 1.0)

Application Properties

Services use application.yml for configuration.

Observability Configuration Files

The project includes examples of pre-configured files for the observability stack:

Grafana

• observability/grafana/provisioning/datasources/datasources.yml - Auto-configures Prometheus, Loki, and Zipkin data sources
• observability/grafana/provisioning/dashboards/notification-logs-dashboard.json - Service logs dashboard with filtering
• observability/grafana/provisioning/dashboards/api-flow-monitoring.json - Real-time API flow monitoring dashboard

Prometheus

• observability/prometheus.yml - Scrape configuration for Spring Boot Actuator endpoints

Loki

• observability/loki-config.yml - Log storage and querying configuration

Promtail

• observability/promtail-config.yml - Log shipping configuration with Spring Boot log parsing