Dynamic KPI Dashboard 📊

Transform your data analytics workflow from manual frontend development to dynamic, configuration-driven dashboards

This project provides a scalable microservices architecture to connect Apache Superset to MongoDB via Trino, enabling dynamic dashboards without manual frontend coding.

🎯 Goals & Value Proposition

Business Problem Solved

Before: Each KPI requires ~1 day of Angular development
After: KPIs created instantly through configuration
Result: 10x faster dashboard creation, zero frontend dependencies

Technical Goals

✅ Zero-code KPI creation for non-technical users
✅ Scalable architecture supporting unlimited metrics
✅ Real-time dashboards with automatic data refresh
✅ Enterprise-grade monitoring and code quality

🏗️ Architecture Overview

Microservices Stack

┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│   Superset  │◄───┤    Trino    │◄───┤   MongoDB   │
│ (Frontend)  │    │ (Query Eng) │    │ (Data Lake) │
└─────────────┘    └─────────────┘    └─────────────┘
       │                   │                   │
       ▼                   ▼                   ▼
  Port 8088           Port 8080           Port 27017

Data Flow

MongoDB: Stores KPI definitions + business data
Trino: Distributed query engine for MongoDB
Superset: Web-based visualization platform
SonarQube: Code quality monitoring
Docker: Container orchestration

🚀 Quick Start

1. Start All Services

# Launch the entire stack
docker-compose up -d

# Check service status
docker-compose ps

2. Generate Sample Data

# Generate financial data for testing
docker-compose run --rm data-generator

3. Access Applications

Service	URL	Credentials
Superset	http://localhost:8088	`admin` / `admin`
Trino UI	http://localhost:8080	No auth required
SonarQube	http://localhost:9000	`admin` / `admin`
MongoDB	localhost:27017	`admin` / `admin`

� CI/CD Pipeline

GitLab CI/CD Stages

┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│  Validate   │───▶│    Test     │───▶│   Quality   │───▶│    Build    │───▶│   Deploy    │
│ • Lint Code │    │ • Unit Tests│    │ • SonarQube │    │ • Docker    │    │ • Staging   │
│ • Dockerfile│    │ • Integration│    │ • Coverage  │    │ • Registry  │    │ • Production│
└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘

Pipeline Features

✅ Automated Code Quality - Linting, formatting, SonarQube analysis
✅ Comprehensive Testing - Unit tests, integration tests, coverage reports
✅ Docker Image Building - Automated image builds and registry push
✅ Multi-Environment Deployment - Staging and production environments
✅ Performance Testing - Locust-based load testing
✅ Manual Gates - Production deployments require manual approval

Required GitLab Variables

Set these in GitLab → Settings → CI/CD → Variables:

SONAR_HOST_URL=https://your-sonarqube-instance.com
SONAR_TOKEN=your-sonarqube-token
CI_REGISTRY_USER=your-registry-username
CI_REGISTRY_PASSWORD=your-registry-password

�🔧 Configuration & Setup

Connect Superset to MongoDB

Go to Superset → Settings → Database Connections
Add new connection:
```
URI: trino://trino@trino:8080/mongodb
```

Sample KPI Definition

Store dynamic KPI configs in MongoDB:

{
  "kpi_id": "revenue_growth",
  "title": "Monthly Revenue Growth",
  "type": "line_chart",
  "query": "SELECT date, SUM(close * volume) as revenue FROM mongodb.financial_data.stock_prices GROUP BY date ORDER BY date",
  "visualization": {
    "format": "currency",
    "refresh_rate": "5min",
    "chart_type": "line"
  },
  "permissions": ["analyst", "manager", "admin"]
}

📈 Code Quality & Testing

SonarQube Analysis

# Start SonarQube
docker-compose up -d sonarqube sonar_postgres

# Run code analysis
docker-compose run --rm sonar-scanner sonar-scanner

# View results at http://localhost:9000

Load Testing with Locust

# Install Locust
pip install locust

# Run performance tests
locust -f src/test/test_superset.py --host=http://localhost:8088

Current Quality Metrics

✅ 0 Bugs detected
✅ 0 Vulnerabilities found
✅ 0 Code Smells identified
⚠️ 1 Encoding Warning (minor)

📊 Sample Queries & Use Cases

Financial Analytics

-- Stock performance by sector
SELECT 
    sector,
    AVG(close) as avg_price,
    SUM(volume) as total_volume
FROM mongodb.financial_data.stock_prices 
WHERE date >= current_date - interval '30' day
GROUP BY sector;

Real-time Monitoring

-- Recent trading activity
SELECT 
    symbol,
    company_name,
    close,
    volume,
    date
FROM mongodb.financial_data.stock_prices 
ORDER BY date DESC 
LIMIT 100;

🔧 Development & Deployment

Project Structure

📁 Dynamic-kpi-dashboard/
├── 📁 src/
│   ├── 📁 main/           # Business logic
│   └── 📁 test/           # Test suites
├── 📁 config/
│   ├── 📁 superset/       # Superset configs
│   └── 📁 trino/          # Trino catalogs
├── 📁 docs/               # Documentation
├── 📁 assets/             # Architecture diagrams
├── 🐳 docker-compose.yml  # Service orchestration
├── 🐳 Dockerfile          # Data generator image
├── 📊 sonar-project.properties  # Code quality config
└── 📋 requirements.txt    # Python dependencies

Technology Stack

Backend: Python 3.11, PyMongo
Database: MongoDB, PostgreSQL
Analytics: Apache Superset, Trino
Quality: SonarQube
DevOps: Docker, Docker Compose

🚦 Monitoring & Observability

Health Checks

# Check all services
docker-compose ps

# View logs
docker-compose logs -f superset
docker-compose logs -f trino
docker-compose logs -f mongodb

Performance Metrics

Superset Response Time: < 2s for dashboards
Trino Query Performance: < 5s for complex aggregations
MongoDB Throughput: 1M+ documents/second

🔮 Roadmap & Future Enhancements

Phase 1 ✅ (Current)

Phase 2 🚧 (In Progress)

KPI Definition Admin Panel
Automated dashboard creation via Superset API
Role-based access control
Real-time data streaming

Phase 3 📋 (Planned)

Advanced visualization templates
Machine learning integration
Multi-tenant support
Enterprise SSO integration

Development Setup

# Clone repository
git clone <repository-url>
cd Dynamic-kpi-dashboard

# Install dependencies
pip install -r requirements.txt

# Run tests
python -m pytest src/test/

# Run code quality checks
docker-compose run --rm sonar-scanner sonar-scanner

Supported Versions

Superset: 3.0.0
Trino: 443
MongoDB: Latest
Python: 3.11+

📝 License & Support

License: MIT License
Maintainer: Smart Conseil Team
Support: Create an issue for bugs or feature requests

🏆 Success Metrics

Metric	Before	After	Improvement
KPI Creation Time	1 day	5 minutes	99% faster
Developer Dependency	High	Zero	100% reduction
Maintenance Overhead	Manual	Automated	Eliminated
Scalability	Limited	Unlimited	∞ growth

🔒 Security Notice

This repository contains only development configurations with default credentials.

For production deployments:

Change all default passwords in docker-compose.yml
Set secure SUPERSET_SECRET_KEY
Use proper database credentials
Configure SonarQube token via SONAR_TOKEN environment variable
Review and update all security settings

Never commit:

Production credentials
API keys or tokens
Personal access tokens
SSL certificates or private keys

Name		Name	Last commit message	Last commit date
Latest commit History 21 Commits
assets		assets
config		config
docs		docs
src		src
.env.template		.env.template
.gitignore		.gitignore
.gitlab-ci.yml		.gitlab-ci.yml
Dockerfile		Dockerfile
README.md		README.md
docker-compose.yml		docker-compose.yml
requirements.txt		requirements.txt
sonar-project.properties		sonar-project.properties

Ibrahimghali/Dynamic-kpi-dashboard

Folders and files

Latest commit

History

Repository files navigation