Kafka Outbox with DDD & Kafka Events

This repository is a reference implementation of the Outbox pattern combined with Domain-Driven Design (DDD) and Kafka-based event streaming.

It is primarily intended as a personal/portfolio project to showcase:

• Cleanly modelled domain events
• Reliable event publication using the Outbox pattern
• Kafka producers/consumers with Schema Registry
• Module boundaries following DDD principles

---

Tech stack

• Runtime: Node.js, TypeScript
• Web framework: Express
• Persistence: MongoDB (single-node replica set)
• Messaging: Kafka + Zookeeper + Confluent Schema Registry
• Background processing / infra: Redis, BullMQ
• Containerisation: Docker & docker-compose
• Other: tsyringe (DI), winston (logging), Joi (validation)

---

High-level architecture

• DDD modules (e.g. session) encapsulate their own:
  • Aggregates
  • Domain services
  • Domain events
• Domain events extend a base DomainEvent and are the single source of truth for what happened in the domain.
• Outbox pattern:
  • When something meaningful happens in the domain (e.g. a session is created), a domain event is recorded into an Outbox collection in MongoDB within the same transaction.
  • A dedicated Outbox worker (OutboxEventWorker) periodically polls the outbox table, converts stored rows into domain events, and publishes them to Kafka via an IEventBus abstraction.
  • On success, events are marked as processed; on failure, they are marked failed with exponential backoff metadata for retries.
• Kafka integration:
  • KafkaEventBus implements IEventBus and is responsible for publishing domain events to Kafka topics, using Confluent Schema Registry to encode Avro payloads.
  • KafkaEventConsumer handles consuming events from Kafka, deserialising them via Schema Registry and dispatching them to subscribed IEventHandlers.
  • A ProcessedEvent store is used to ensure idempotent consumption and avoid reprocessing duplicates.
• Session module example:
  • SessionCreatedEvent is raised when a new session is created.
  • It is persisted to the outbox.
  • The Outbox worker publishes it to the Kafka topic session-events.v1.
  • The Kafka consumer subscribes to the topics defined in EventTopicMapper and routes events to the appropriate handlers.

---

Key patterns implemented

• Outbox pattern
  • Decouples domain writes from asynchronous message publishing.
  • Provides at-least-once delivery guarantees with controlled retries.
• Domain events as first-class citizens
  • Each event has: eventId, aggregateId, aggregateType, eventName, occurredOn, version, and a typed payload.
• Schema Registry + Avro
  • Each event version is associated with a registered schema.
  • Encoding/decoding is delegated to the Schema Registry client to keep producers/consumers strongly typed.
• Idempotent consumers
  • Each consumed event is recorded in a ProcessedEvent collection.
  • Before handling, the consumer checks if the event has already been processed and skips duplicates.

---

Running the project locally (Docker)

1. Prerequisites

• Docker and docker-compose installed.
• Basic familiarity with Kafka and MongoDB is helpful, but not required.

2. Environment variables

The development environment variables for Docker are defined in:

• env/.env.docker.dev

Adjust values like:

• API_HOST_PORT / API_CONTAINER_PORT
• DB_PORT
• MongoDB credentials

…to match your local preferences if needed.

3. Start the stack

From the project root:

docker compose -f docker-compose.dev.yml up --build

This will start:

• App (server) – Express API + Outbox worker and Kafka wiring
• MongoDB – with replica set configuration for transactional support
• Redis – for BullMQ and caching
• Zookeeper + Kafka
• Schema Registry
• Kafka UI on port 8080

Once everything is up:

• The API will be reachable on http://localhost:${API_HOST_PORT} (see your .env).
• Kafka UI will be reachable on http://localhost:8080.

---

Server scripts (from server/)

Inside the server folder:

# Development server (without Docker, if you want)
yarn dev

# Build TypeScript
yarn build

# Start compiled server
yarn start

# Seed Kafka topics in Schema Registry / Kafka
yarn seed:topics

When using Docker (docker-compose.dev.yml), the app service runs the server inside the container, using env/.env.docker.dev.

---

Event flow example: SessionCreated

1. A new session is created in the system.
2. The domain raises a SessionCreatedEvent which extends DomainEvent.
3. The event is persisted to the Outbox collection in MongoDB, within the same unit of work as the session write.
4. OutboxEventWorker periodically:
   • Claims a batch of pending outbox events (claimPending).
   • Maps them back to concrete domain event instances (OutboxMapper).
   • Publishes them via KafkaEventBus to the session-events.v1 Kafka topic.
   • Marks successful events as processed, or failed with an exponential backoff schedule.
5. KafkaEventConsumer:
   • Subscribes to topics defined in EventTopicMapper (e.g. session-events.v1).
   • Decodes messages using Schema Registry.
   • Checks the ProcessedEvent store for idempotency.
   • Dispatches each event to the corresponding registered handlers.

This gives you a realistic, production-flavoured end-to-end pipeline from domain event → persistent outbox → Kafka → consumer handlers, while keeping the domain clean and well-structured.

---

Why this project exists

This codebase is meant to be a practical, end-to-end example you can point to that demonstrates:

• Understanding of DDD and modular design.
• Ability to implement reliable, event-driven architectures with Kafka.
• Comfort with Docker-based local infrastructure (Kafka, Mongo, Redis, Schema Registry).

Feel free to browse through the server/src structure, especially:

• shared/domain/events – core event abstractions and mappings.
• shared/infrastructure/messaging/kafka – Kafka client, event bus, consumers, Schema Registry integration.
• shared/application/worker – Outbox worker implementation.
• modules/session – sample bounded context and event definitions.