⭐ Please Star this Repo if You Enjoy It! ⭐
Production-grade threat detection platform processing 10,000+ logs/second with ML-powered analysis
TLDR: Token Expires Every 24hs need auth token to acess! Changed so anyone can acess without token for now but version limited
- Architecture Guide — System design details
- Deployment Guide — Production deployment
- ML Models — Model training and evaluation
An enterprise-grade security operations platform that detects threats in real-time using machine learning and rule-based analysis. Built with microservices architecture for scalability and production deployment.
Use Cases:
- Security Operations Center (SOC) monitoring
- Threat intelligence and incident response
- Compliance monitoring (PCI-DSS, SOC2)
- Cloud security event analysis
- 🤖 ML-Powered Detection - IsolationForest + RandomForest models (92% accuracy)
- ⚡ High Performance - Processes 10,000+ logs/second with <150ms latency
- 📊 Real-Time Dashboard - Live threat monitoring with Streamlit
- 🔌 Multiple Ingestion Sources - HTTP, Syslog, Kafka, File watching, Packet capture
- 📈 Full Observability - Prometheus metrics, Grafana dashboards
- 🐳 Production Ready - Docker Compose, health checks, auto-scaling
# Submit a SQL injection attempt
curl -X POST "https://api.your-demo.com/api/v1/threats/analyze" \
-H "Authorization: Bearer $TOKEN" \
-d '{"log_data": "SELECT * FROM users WHERE id=1 OR 1=1--", "source_ip": "10.0.0.1"}'
# Response (92.5 risk score - CRITICAL)
{
"threat_id": "abc123",
"risk_score": 92.5,
"threat_type": "SQL_INJECTION",
"severity": "CRITICAL",
"confidence": 0.94
}┌──────────────┐ ┌───────────────┐ ┌──────────────┐
│ Ingestion │───→│ Redis Queue │───→│ Detection │
│ Service │ │ (10K/sec) │ │ Engine │
│ │ │ │ │ (ML + Rules)│
└──────────────┘ └───────────────┘ └──────────────┘
↓ Multiple ↓
Sources: ┌──────────────┐
• HTTP API │ PostgreSQL │
• Syslog │ (Storage) │
• Kafka └──────────────┘
• File Watch ↓
• PCAP ┌──────────────┐
│ Dashboard │
│ (Streamlit) │
└──────────────┘
- Backend: FastAPI, Python 3.11+
- ML: scikit-learn (IsolationForest, RandomForest)
- Queue: Redis (batching, caching)
- Database: PostgreSQL (threat storage)
- Monitoring: Prometheus, Grafana
- Ingestion: Scapy, Kafka, Syslog
- Frontend: Streamlit with Plotly
- Python 3.11+
- Docker & Docker Compose
- Redis
- PostgreSQL
git clone https://github.com/Shaid-T/Enterprise-Security-Platform.git
cd Enterprise-Security-Platform
# Create virtual environment
python3.11 -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt# Start infrastructure
docker-compose up -d redis postgres
# Initialize database
psql -h localhost -U threat_user -d security_platform -f init-db.sql
# Start API Gateway
uvicorn services.fastapi_app:app --reload --port 8000 &
# Start Detection Service
python services/detection_service.py &
# Start Ingestion Service
uvicorn services.ingestion_service:app --reload --port 9000 &
# Start Dashboard
npm run dev
## 📊 Performance Benchmarks
| Metric | Result | Target |
|--------|--------|--------|
| **Throughput** | 750 req/sec | 500+ |
| **Detection Latency (p95)** | 145ms | <200ms |
| **Ingestion Rate** | 10,000 logs/sec | 5,000+ |
| **ML Accuracy** | 92.3% | >90% |
| **Queue Processing** | 1,200 jobs/sec | 1,000+ |
*Tested on: 4 vCPU, 8GB RAM*
---
## 🎯 Detection Capabilities
### Threat Types Detected
- **SQL Injection** - Pattern matching + ML anomaly detection
- **Cross-Site Scripting (XSS)** - Script tag and event handler detection
- **Command Injection** - Shell command pattern analysis
- **Path Traversal** - Directory traversal attempt detection
- **Brute Force** - Failed login pattern recognition
- **Malware Signatures** - Code execution pattern matching
- **LDAP Injection** - LDAP query manipulation detection
### Detection Rules
- 8 built-in rules with configurable severity
- Custom rule support via API
- ML-based anomaly detection for zero-day threats
- Confidence scoring (0-1 scale)
---
## 📈 Real-World Usage
### Example: SOC Monitoring
```python
# Ingest 10,000 logs from various sources
POST /ingest/http # HTTP API
UDP 5140 # Syslog
/logs/*.log # File watcher
kafka://logs # Kafka consumer
# Detection Engine processes in parallel
# Critical threats trigger immediate alerts
# Dashboard shows real-time statistics# Query historical threats
GET /api/v1/threats/?severity=CRITICAL&hours=24
# Export for forensics
GET /api/v1/threats/export?format=csv
# Block attacking IPs
POST /api/v1/blocks {"ip": "10.0.0.50"}# Redis
REDIS_URL=redis://localhost:6379
REDIS_MAX_CONNECTIONS=100
# Database
DATABASE_URL=postgresql://user:pass@localhost:5432/security_platform
# Detection
ALERT_THRESHOLD=70.0
ML_WEIGHT_ISOLATION=0.5
ML_WEIGHT_RANDOM_FOREST=0.5
# Ingestion
BATCH_SIZE=100
BATCH_TIMEOUT_MS=500# Scale detection workers
docker-compose up -d --scale detection-service=5
# Adjust queue batch size
BATCH_SIZE=200 # Process 200 logs per batch# Unit tests
pytest tests/ -v --cov
# Load testing (Apache Bench)
ab -n 1000 -c 50 \
-H "Authorization: Bearer $TOKEN" \
-p payload.json \
http://localhost:8000/api/v1/threats/analyze
# Integration tests
pytest tests/integration/ -v# Submit 1000 test threats
python scripts/generate_test_threats.py --count 1000- API Documentation - Interactive API docs
- Architecture Guide - System design details
- Deployment Guide - Production deployment
- ML Models - Model training and evaluation
docker-compose up -dkubectl apply -f k8s/See DEPLOYMENT.md for detailed instructions.
Contributions welcome! Please read CONTRIBUTING.md first.
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
This project is licensed under the MIT License - see LICENSE file for details.
Built as part of the Advanced Security Infrastructure Roadmap (Phase 6)
⭐ Star this repo if you find it useful!
Built with ❤️ for the security community