Log Aggregation System

A production-style log aggregation and mini-monitoring system built with TypeScript, Kafka, TimescaleDB, Prometheus, Grafana, and Jaeger.

Architecture

[Services / SDK] → POST /logs → [Ingestor :3000] → Kafka (logs topic)
                                                          │
                              ┌───────────────────────────┤
                              │                           │
                       [Log Worker]              [Metrics Worker]
                              │                           │
                       TimescaleDB               Prometheus counters
                              │                           │
                              └──────────┬────────────────┘
                                         │
                                   [Grafana :3001]
                                         │
                                  [Alert Worker] → Kafka (alerts topic)

[Jaeger :16686] ← OTel spans from services (external, not custom-built)

Logs vs Metrics vs Traces

Concern	What	Storage	Tool
Logs	Discrete text events	TimescaleDB	This service
Metrics	Aggregated numbers over time	Prometheus	This service
Traces	Distributed request flow	Jaeger	External (OTel)

This distinction matters in interviews: logs are queryable events, metrics are pre-aggregated numbers, traces are causally linked spans across services.

Quick Start

cp .env.example .env
docker compose up -d
npm install
npm run dev:ingestor       # http://localhost:3000
npm run dev:log-worker
npm run dev:metrics-worker
npm run dev:alert-worker

API

POST /logs

{
  "service": "my-service",
  "level": "error",
  "message": "something went wrong",
  "timestamp": "2026-07-01T12:00:00.000Z",
  "traceId": "optional-trace-id"
}

Returns 201 Created on success.

GET /metrics

Prometheus metrics endpoint (scraped by Prometheus every 15s).

GET /health

Returns { "status": "ok" }.

Client SDK

Other services use the SDK to push logs without hand-crafting HTTP calls:

import { LogAggregatorClient } from './src/sdk';

const client = new LogAggregatorClient({ url: 'http://localhost:3000' });

await client.send({
  service: 'my-service',
  level: 'error',
  message: 'Payment failed',
  timestamp: new Date().toISOString(),
});

Grafana Dashboard

Open http://localhost:3001 (admin / admin). The dashboard is auto-provisioned with:

• Error Rate Over Time — rate(worker_errors_total[5m]) per service
• Log Volume per Service — rate(worker_logs_processed_total[5m]) per service/level
• Recent Error Logs — live table query from TimescaleDB
• Alert rule — fires when error rate > 5% (checks every 1 minute)

Traces (Jaeger)

Open http://localhost:16686. Services export OTel spans to port 4318. Trace storage and search are handled entirely by Jaeger — this project focused its custom build on logs and metrics aggregation, using Jaeger as an integrated component rather than rebuilding trace storage from scratch.

Alert Worker

The alert worker is a custom rule engine (not just a Grafana alert):

1. Queries Prometheus for rate(worker_errors_total[5m]) on a configurable interval
2. Compares each result against ALERT_ERROR_RATE_THRESHOLD
3. Publishes AlertEvent to the Kafka alerts topic for downstream consumers

This demonstrates alerting as a system concept, separate from dashboard-level alerts.

Built vs Integrated

Component	Built	Integrated
Log ingestion endpoint	✅
Kafka pipeline	✅
TimescaleDB storage	✅
Prometheus metrics	✅
Alert rule engine	✅
Client SDK	✅
Grafana dashboards		✅
Trace storage		✅ (Jaeger)

Knowing this boundary is a real interview talking point — production systems integrate Grafana and Jaeger rather than rebuilding them.

What I'd Change at Scale

• Log search: Replace Postgres ILIKE with Elasticsearch for full-text search at > 10M logs/day
• Trace sampling: Add head-based sampling before Jaeger to avoid 100% trace overhead at high throughput
• Prometheus federation: Add a central federation layer when scraping > 50 services
• Kafka partitioning: Partition the logs topic by service to preserve per-service ordering and enable parallel worker scaling
• Alert delivery: Use PagerDuty/Opsgenie instead of Kafka topic; the custom engine shows the concept, not the production answer

Tests

npm test              # unit tests (30 tests, ~98% coverage)
npm run test:coverage # with coverage report

# Integration test (requires docker compose up -d):
npx jest tests/integration/ingestion.test.ts --no-coverage