🌊 LogStream - Real-time Log Aggregation & Monitoring System

A beautiful, production-ready log monitoring system with AI-powered insights using Groq, real-time WebSocket streaming, and a stunning React dashboard.

🎯 Features

📸 Dashboard Preview

🚀 Overview

🎯 Filtering & Metrics

📡 Live Activity Feed

📡 Live Alerts

📡 Live Activity Feed Filtered

Backend

⚡ Real-time Log Processing - Asynchronous queue-based processing with Bull & Redis
🤖 AI-Powered Insights - Groq AI analyzes error logs and provides root cause analysis
📊 Metrics & Analytics - Real-time metrics calculation and historical tracking
🔌 WebSocket Streaming - Live log broadcasting to connected clients
💾 PostgreSQL Storage - Persistent log storage with efficient indexing
🚨 Smart Alerting - Configurable alert triggers for error rates and critical keywords
📡 REST API - Complete API for log ingestion and retrieval
🐳 Fully Dockerized - One command setup with Docker Compose

Frontend

🎨 Beautiful UI - Glassmorphism design with smooth animations
✨ Framer Motion - Smooth transitions and micro-interactions
📈 Live Charts - Real-time metrics visualization with Recharts
🔍 Advanced Filtering - Search and filter by level, source, and time
🤖 AI Insights Display - Beautiful presentation of Groq AI analysis
🌐 Real-time Updates - WebSocket connection for instant log streaming
🎯 Toast Notifications - Non-intrusive alerts for errors and insights

🏗️ Architecture

[Clients] → [REST API] → [Bull Queue + Redis] → [Workers] → [PostgreSQL]
                                                      ↓
                                            [WebSocket Server] → [React Dashboard]
                                                      ↓
                                                 [Groq AI] → [AI Insights]

📋 Prerequisites

Docker Setup (Recommended):

Docker Desktop - Download here
Groq API Key (free from console.groq.com)

Manual Setup:

Node.js 20+ and npm
PostgreSQL 15+
Redis 7+
Groq API Key

🐳 Quick Start with Docker (Recommended)

This is the easiest way to run LogStream — no need to install or run PostgreSQL or Redis separately.

1. Clone the repo

git clone https://github.com/your-username/logstream-backend.git
cd logstream-backend

2. Set up environment

cp .env.example .env

Edit .env and fill in your values:

POSTGRES_USER=your_db_user
POSTGRES_PASSWORD=your_db_password
GROQ_API_KEY=gsk_your_groq_api_key_here

Make sure REDIS_HOST=redis and POSTGRES_HOST=db in your .env (already set in .env.example)

3. Start everything

docker compose up --build

That's it! All three services (app, PostgreSQL, Redis) start automatically.

Access the API

REST API: http://localhost:3001/api
Health Check: http://localhost:3001/api/health
WebSocket: ws://localhost:3001

Stop

docker compose down

Stop and clear all data

docker compose down -v

🖥️ Running Full Stack (Backend + Frontend)

Frontend repo: https://github.com/dishamurthy/logstream-dashboard

Step 1 — Start backend first (this repo):

cd logstream-backend
cp .env.example .env   # fill in your values
docker compose up --build

Step 2 — In a new terminal, start frontend:

cd logstream-dashboard
docker compose up --build

Step 3 — Open browser:

Dashboard → http://localhost:5173
API → http://localhost:3001/api

🖥️ Manual Setup (Mac)

1. Backend Setup

cd logstream-backend

Install Dependencies

npm install express bull ioredis pg socket.io cors dotenv winston joi uuid axios
npm install -D typescript ts-node-dev @types/express @types/node @types/bull @types/cors @types/uuid jest @types/jest

Configure Environment

Create .env file:

# Server Configuration
PORT=3000
NODE_ENV=development

# Redis Configuration
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=

# PostgreSQL Configuration
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_DB=logstream
POSTGRES_USER=your_username
POSTGRES_PASSWORD=your_password

# WebSocket Configuration
WS_PORT=3001
WS_CORS_ORIGIN=http://localhost:5173

# Queue Configuration
QUEUE_CONCURRENCY=5
QUEUE_MAX_RETRIES=3

# Log Retention (in days)
LOG_RETENTION_DAYS=30

# Alert Thresholds
ERROR_RATE_THRESHOLD=10
LATENCY_THRESHOLD_MS=1000

# Groq AI Configuration
GROQ_API_KEY=gsk_your_groq_api_key_here
GROQ_MODEL=llama-3.3-70b-versatile
GROQ_API_URL=https://api.groq.com/openai/v1/chat/completions
GROQ_ENABLED=true

Setup Database & Redis (Mac)

brew install postgresql@15 redis
brew services start postgresql@15
brew services start redis
createdb logstream

Start Backend

# Terminal 1: API Server
npm run dev

# Terminal 2: Worker Process
npm run worker

2. Frontend Setup

cd ~/Desktop
npm create vite@latest logstream-dashboard -- --template react-ts
cd logstream-dashboard
npm install socket.io-client recharts framer-motion lucide-react react-hot-toast date-fns
npm install -D tailwindcss postcss autoprefixer
npx tailwindcss init -p
npm run dev

Open browser at: http://localhost:5173

📡 API Endpoints

Health Check

GET /api/health

Ingest Single Log

POST /api/logs
Content-Type: application/json

{
  "level": "error",
  "message": "Database connection failed",
  "source": "api_gateway",
  "userId": "user-123",
  "metadata": { "ip": "192.168.1.1" }
}

Ingest Batch Logs

POST /api/logs/batch
Content-Type: application/json

{
  "logs": [
    { "level": "info", "message": "User logged in", "source": "service" },
    { "level": "error", "message": "Payment failed", "source": "service" }
  ]
}

Retrieve Logs

GET /api/logs?level=error&page=1&pageSize=50
GET /api/logs?source=api_gateway&startTime=2024-01-01T00:00:00Z

Get Metrics

GET /api/metrics?window=5

Queue Statistics

GET /api/queue/stats

🔌 WebSocket Events

const socket = io('ws://localhost:3001');

socket.on('log:new', (event) => console.log('New log:', event.data));
socket.on('metrics:update', (event) => console.log('Metrics:', event.data));
socket.on('alert:triggered', (event) => console.log('Alert:', event.data));

socket.emit('subscribe:level', 'error');
socket.emit('subscribe:source', 'api_gateway');

🤖 Groq AI Integration

Error/Fatal logs are automatically sent to Groq AI which returns:

Summary: Brief description of the issue
Root Cause: Likely cause of the error
Suggested Fix: How to resolve the issue
Severity: low, medium, high, or critical

Get Your Groq API Key

Visit console.groq.com
Sign up / Login → API Keys → Create API Key
Add to .env as GROQ_API_KEY=gsk_...

📊 Database Schema

CREATE TABLE logs (
  id VARCHAR(36) PRIMARY KEY,
  level VARCHAR(10) NOT NULL,
  message TEXT NOT NULL,
  source VARCHAR(50) NOT NULL,
  timestamp TIMESTAMPTZ NOT NULL,
  parsed_at TIMESTAMPTZ NOT NULL,
  metadata JSONB,
  user_id VARCHAR(100),
  request_id VARCHAR(100),
  session_id VARCHAR(100),
  tags TEXT[],
  stack_trace TEXT,
  context JSONB,
  ai_insight JSONB
);

🧪 Testing

node test-client.js single       # Send single log
node test-client.js batch 20     # Send batch of 20 logs
node test-client.js simulate 120000  # Simulate traffic for 2 minutes
node test-client.js errors       # Send error logs (triggers AI analysis)

# Manual curl test
curl -X POST http://localhost:3001/api/logs \
  -H "Content-Type: application/json" \
  -d '{"level": "error", "message": "Payment processing failed", "source": "service"}'

📁 Project Structure

logstream-backend/
├── src/
│   ├── api/routes.ts
│   ├── workers/logProcessor.ts
│   ├── config/index.ts
│   ├── types/index.ts
│   ├── utils/
│   │   ├── database.ts
│   │   ├── queue.ts
│   │   ├── websocket.ts
│   │   ├── validation.ts
│   │   └── groq.ts
│   └── index.ts
├── Dockerfile
├── docker-compose.yml
├── .dockerignore
├── .env.example          # Safe to commit - no real secrets
├── .env                  # Never committed - your real secrets
├── .gitignore
├── package.json
└── tsconfig.json

🚨 Troubleshooting

Docker: Port already in use

lsof -ti:3001 | xargs kill -9
docker compose down
docker compose up --build

Docker: Database errors on startup

docker compose down -v   # clears volumes
docker compose up --build

Manual: Redis connection failed

brew services restart redis
redis-cli ping  # Should return PONG

Manual: PostgreSQL connection failed

brew services restart postgresql@15
psql -l

Groq API errors

Check your API key in .env
Verify GROQ_ENABLED=true
Check quota at console.groq.com

🔒 Security

.env is gitignored — never commit real secrets
Use .env.example with dummy values for sharing
Add authentication before production deployment
Implement rate limiting on API endpoints

🤝 Contributing

Fork the repository
Create feature branch: git checkout -b feature/amazing-feature
Commit: git commit -m 'Add amazing feature'
Push: git push origin feature/amazing-feature
Open a Pull Request

📝 License

MIT License - feel free to use for personal or commercial purposes.

Built with ❤️ using React, TypeScript, Node.js, PostgreSQL, Redis, Groq AI, and Docker

⭐ Star this repo if you find it useful!

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
assets		assets
src		src
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
Dockerfile		Dockerfile
README.md		README.md
docker-compose.yml		docker-compose.yml
package-lock.json		package-lock.json
package.json		package.json
test-client.js		test-client.js
tsconfig.json		tsconfig.json

dishamurthy/logstream-backend

Folders and files

Latest commit

History

Repository files navigation