🛡️ Distributed Log Sentinel

"Systems fall. Logs survive." — DevOps proverb

A distributed logging service built with Python and Flask. Multiple simulated services generate and ship logs over HTTP to a central server that authenticates, stores, and exposes them for querying — with optional filters, stats, and automatic cleanup.

Built as part of the Penguin Academy backend challenge series.

---

📋 Table of Contents

• Overview
• Architecture
• Project Structure
• Features
• Getting Started
• API Reference
• Authentication
• Simulated Clients
• Bonus Features
• Tech Stack

---

Overview

Distributed Log Sentinel solves the classic "nothing was logged, so technically nothing happened" problem. It consists of two parts:

• Central Log Server — a Flask HTTP server that receives, validates, stores, and serves logs from a SQLite database.
• Simulated Service Clients — scripts that impersonate different microservices, generate realistic-looking fake logs, and fire them at the server via POST requests with token authentication.

---

Architecture

┌──────────────────────────────────────────────────────────────┐
│                        clients/                              │
│                                                              │
│                       services.py                            │
│               (simulates multiple services)                  │
│                           │                                  │
│              POST /logs   │   Authorization: Token <token>   │
└──────────────────────────┬───────────────────────────────────┘
                           │
                           ▼
┌──────────────────────────────────────────────────────────────┐
│                        server/                               │
│                                                              │
│   app.py  →  Token Validation  →  database.py  →  SQLite     │
│                                                              │
│   POST /logs      — receive & store logs                     │
│   GET  /logs      — query logs with filters                  │
│   GET  /stats     — metrics per service & severity           │
│   DELETE /logs    — cleanup old logs by date                 │
└──────────────────────────────────────────────────────────────┘

---

Project Structure

distributed-log-sentinel/
├── server/
│   ├── app.py              # Flask application & all route definitions
│   └── database.py         # SQLite setup and query helpers
├── clients/
│   └── services.py         # Simulates multiple services sending logs
├── requirements.txt        # Python dependencies
├── .gitignore
└── README.md

---

Features

Feature	Status
Multiple simulated service clients	✅
POST /logs — receive & persist logs	✅
GET /logs — query with date filters	✅
Token-based authentication per service	✅
SQLite persistence	✅
Batch log ingestion (multiple logs per request)	✅
Clear HTTP responses and error messages	✅
GET /stats — metrics by service and severity	✅ Bonus
DELETE /logs?before=... — auto-cleanup old logs	✅ Bonus
Periodic log sending via threading	✅ Bonus

---

Getting Started

Prerequisites

• Python 3.8+
• pip

Installation

# Clone the repository
git clone https://github.com/IvanOcampos/distributed-log-sentinel.git
cd distributed-log-sentinel

# Create and activate a virtual environment
python -m venv venv

# On Windows
venv\Scripts\activate

# On macOS/Linux
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

Running the Server

cd server
python app.py

The server will start on http://localhost:5000.

Running the Clients

Open a new terminal (with the virtual environment active) and run any of the simulated service scripts:

# Run clients
python clients/services.py

---

API Reference

POST /logs

Receives one or more logs from an authenticated service.

Headers:

Authorization: Token <your_token_here>
Content-Type: application/json

Body (single log):

{
  "timestamp": "2025-04-01T12:00:00",
  "service": "auth-service",
  "severity": "ERROR",
  "message": "Failed login attempt for user ghost@example.com"
}

Body (batch — multiple logs):

[
  {
    "timestamp": "2025-04-01T12:00:00",
    "service": "auth-service",
    "severity": "INFO",
    "message": "User logged in successfully"
  },
  {
    "timestamp": "2025-04-01T12:00:05",
    "service": "auth-service",
    "severity": "DEBUG",
    "message": "Session token generated"
  }
]

Response 201:

{ "message": "Log(s) saved successfully" }

Response 401 (invalid token):

{ "error": "Who are you?" }

---

GET /logs

Returns stored logs. Supports optional query parameter filters.

Headers:

Authorization: Token <your_token_here>

Optional Query Parameters:

Parameter	Format	Description
timestamp_start	YYYY-MM-DDTHH:MM:SS	Filter by log event time (start)
timestamp_end	YYYY-MM-DDTHH:MM:SS	Filter by log event time (end)
received_at_start	YYYY-MM-DDTHH:MM:SS	Filter by server ingestion time (start)
received_at_end	YYYY-MM-DDTHH:MM:SS	Filter by server ingestion time (end)
service	string	Filter by service name
severity	INFO | DEBUG | ERROR | WARNING	Filter by severity level

Example:

curl -H "Authorization: Token auth-service-token-001" \
  "http://localhost:5000/logs?severity=ERROR&service=auth-service"

Response 200:

[
  {
    "id": 1,
    "timestamp": "2025-04-01T12:00:00",
    "service": "auth-service",
    "severity": "ERROR",
    "message": "Failed login attempt for user ghost@example.com",
    "received_at": "2025-04-01T12:00:01"
  }
]

---

GET /stats

Returns aggregate metrics across all logs.

Response 200:

{
  "logs_per_service": {
    "auth-service": 120,
    "payment-service": 85,
    "notification-service": 43
  },
  "logs_per_severity": {
    "INFO": 150,
    "DEBUG": 60,
    "ERROR": 30,
    "WARNING": 8
  },
  "last_log_per_service": {
    "auth-service": "2025-04-01T12:05:00",
    "payment-service": "2025-04-01T12:04:30",
    "notification-service": "2025-04-01T11:59:00"
  }
}

---

DELETE /logs?before=<date>

Deletes all logs with a timestamp before the given date. Useful for cleanup.

Example:

curl -X DELETE -H "Authorization: Token auth-service-token-001" \
  "http://localhost:5000/logs?before=2025-01-01T00:00:00"

Response 200:

{ "message": "42 log(s) deleted" }

---

Authentication

Each simulated service has a unique static token. Tokens are validated by the server on every request.

Service	Token
auth-service	auth-service-token-001
payment-service	payment-service-token-002
notification-service	notification-service-token-003

Tokens are sent in the Authorization header using the format:

Authorization: Token <token_value>

If the token is missing or invalid, the server responds with:

{ "error": "Quién sos, bro?" }

Note: Static tokens are used here for simplicity. For a production setup, consider replacing them with JWT (JSON Web Tokens).

---

Simulated Clients

Each client script in clients/ simulates a real microservice by:

1. Generating fake but realistic log entries (randomized severity and messages).
2. Attaching a timestamp of the exact moment the "event" occurred.
3. Sending logs to the central server via POST /logs with its unique token.
4. Optionally looping with a delay to simulate continuous log emission.

Log payload structure:

{
  "timestamp": "2025-04-01T12:00:00",
  "service": "payment-service",
  "severity": "WARNING",
  "message": "Transaction timeout for order #8821 — retrying..."
}

---

Bonus Features

Periodic Log Emission

Clients can be configured to send logs automatically every N seconds using Python's threading module — simulating real background services generating activity continuously.

Auto-Cleanup Endpoint

The DELETE /logs?before=<date> endpoint allows removing stale log data on demand, keeping the database lean.

Stats Dashboard (via /stats)

A lightweight stats endpoint gives a quick snapshot of system activity without querying the full log table.

---

Tech Stack

Layer	Technology
Language	Python 3
Web Framework	Flask
Database	SQLite (via Python's built-in sqlite3)
HTTP Client (clients)	requests
Auth	Static Bearer Tokens
Concurrency (bonus)	threading

---

Testing with curl

# Send a single log
curl -X POST http://localhost:5000/logs \
  -H "Authorization: Token auth-service-token-001" \
  -H "Content-Type: application/json" \
  -d '{"timestamp":"2025-04-01T10:00:00","service":"auth-service","severity":"INFO","message":"User login successful"}'

# Query all ERROR logs
curl -H "Authorization: Token auth-service-token-001" \
  "http://localhost:5000/logs?severity=ERROR"

# Get stats
curl -H "Authorization: Token auth-service-token-001" \
  http://localhost:5000/stats

# Delete logs older than a date
curl -X DELETE -H "Authorization: Token auth-service-token-001" \
  "http://localhost:5000/logs?before=2025-01-01T00:00:00"

---

Developed by Ivan Ocampos