📈 Real-Time Stock Analytics Pipeline

A production-ready streaming analytics system that ingests live stock market data, processes it in real-time, and visualizes insights through an interactive dashboard.

🎯 Project Overview

This project demonstrates enterprise-grade data engineering practices with a focus on:

• Real-time data streaming with Redis Streams
• Technical analysis with 10+ financial indicators
• Interactive visualization with Streamlit
• Dual storage architecture (Redis + SQLite)
• Production-ready code with error handling and logging

Key Features

✨ Real-Time Processing: Sub-100ms latency from data ingestion to visualization
📊 Technical Indicators: RSI, MACD, Bollinger Bands, Moving Averages, and more
🎨 Interactive Dashboard: Live charts with auto-refresh every 2 seconds
🔄 Fault Tolerance: Automatic retry logic and error recovery
💾 Dual Storage: Redis for caching, SQLite for historical data
📈 Multiple Stocks: Track 5+ stocks simultaneously
🔍 Analytics: Price volatility, momentum, trading signals

🏗️ Architecture

┌─────────────────┐
│   Data Source   │  yfinance API (free)
│  (Yahoo Finance)│  Real-time stock prices
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│    Producer     │  Polls API every 2 seconds
│  (src/producer) │  Validates & publishes data
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Redis Streams  │  Message queue & pub/sub
│   (Port 6379)   │  Buffers data stream
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│    Consumer     │  Reads from stream
│  (src/consumer) │  Calculates indicators
└────────┬────────┘
         │
         ├─────────────────┐
         ▼                 ▼
┌─────────────┐   ┌─────────────┐
│ Redis Cache │   │   SQLite    │
│  (Real-time)│   │ (Historical)│
└──────┬──────┘   └──────┬──────┘
       │                 │
       └────────┬────────┘
                ▼
       ┌─────────────────┐
       │   Dashboard     │  Streamlit web app
       │ (Port 8501)     │  Interactive charts
       └─────────────────┘

📋 Technology Stack

Component	Technology	Purpose
Language	Python 3.9+	Core development
Message Queue	Redis Streams	Data streaming
Database	SQLite	Historical storage
Cache	Redis	Real-time data
Dashboard	Streamlit	Web visualization
Charts	Plotly	Interactive graphs
Data Source	yfinance	Stock market API
Analytics	Pandas, NumPy	Data processing
Logging	Loguru	Structured logging

🚀 Quick Start

Prerequisites

• Python 3.9 or higher
• Redis 6.0 or higher
• 4GB RAM minimum
• Internet connection

Installation

1. Clone the repository

git clone https://github.com/yourusername/real-time-stock-analytics.git
cd real-time-stock-analytics

2. Run setup script

chmod +x setup.sh
./setup.sh

This will:

• Check Python and Redis versions
• Create virtual environment
• Install all dependencies
• Initialize database
• Start Redis server
• Create necessary directories

3. Configure environment (optional)

cp .env.example .env
nano .env  # Edit stocks, intervals, etc.

4. Start all services

chmod +x run.sh
./run.sh

This starts:

• Data Producer (fetches stock data)
• Stream Consumer (processes data)
• Dashboard (http://localhost:8501)

Manual Start (Alternative)

If you prefer to start services individually:

# Terminal 1: Start Producer
source venv/bin/activate
python -m src.producer

# Terminal 2: Start Consumer
source venv/bin/activate
python -m src.consumer

# Terminal 3: Start Dashboard
source venv/bin/activate
streamlit run dashboard/app.py

📊 Technical Indicators

The system calculates the following indicators in real-time:

Moving Averages

• SMA (5, 10, 20): Simple Moving Average for trend analysis
• EMA (12, 26): Exponential Moving Average for recent trends

Momentum Indicators

• RSI (14): Relative Strength Index (0-100)
  • < 30: Oversold

  • 70: Overbought

• Momentum (10): 10-period price momentum

Volatility Indicators

• Bollinger Bands: Upper, Middle, Lower bands
• Price Volatility: Standard deviation of returns

Trend Indicators

• MACD: Moving Average Convergence Divergence
• MACD Signal: Signal line for crossovers
• MACD Histogram: Difference between MACD and signal

📁 Project Structure

real-time-stock-analytics/
├── src/                          # Source code
│   ├── __init__.py
│   ├── config.py                 # Configuration management
│   ├── producer.py               # Data ingestion service
│   ├── consumer.py               # Stream processing service
│   ├── analytics.py              # Technical indicators
│   ├── storage.py                # Redis + SQLite operations
│   └── utils.py                  # Helper functions
│
├── dashboard/                    # Streamlit dashboard
│   └── app.py                    # Main dashboard application
│
├── tests/                        # Unit tests
│   ├── test_producer.py
│   ├── test_consumer.py
│   └── test_analytics.py
│
├── data/                         # Data storage
│   └── stocks.db                 # SQLite database
│
├── logs/                         # Application logs
│   └── app.log
│
├── docs/                         # Documentation
│   ├── architecture.md
│   └── screenshots/
│
├── requirements.txt              # Python dependencies
├── setup.sh                      # Setup script
├── run.sh                        # Startup script
├── .env.example                  # Environment template
├── .gitignore
└── README.md

🎨 Dashboard Features

Main View

• Live Status Indicator: Green pulse when system is active
• Real-Time Metrics: Current price, volume, RSI for each stock
• Price Charts: Interactive line charts with moving averages
• Volume Analysis: Color-coded volume bars
• Technical Indicators: RSI and MACD charts

Sidebar Controls

• Stock Selection: Multi-select dropdown for stocks
• Time Range: 1H, 6H, 24H, 7D views
• Indicator Toggles: Show/hide RSI, MACD, Volume
• System Statistics: Data points per stock

Interactive Features

• Auto-refresh every 2 seconds
• Hover tooltips on charts
• Zoom and pan capabilities
• Expandable analytics panels
• Responsive design

🔧 Configuration

Edit .env file to customize:

# Redis Configuration
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_DB=0

# Stock Configuration (comma-separated)
STOCKS=AAPL,GOOGL,MSFT,TSLA,AMZN

# Logging
LOG_LEVEL=INFO

# API Configuration
MAX_RETRIES=3
RETRY_DELAY=5

📈 Usage Examples

Add More Stocks

Edit .env:

STOCKS=AAPL,GOOGL,MSFT,TSLA,AMZN,NVDA,META,NFLX

Restart services:

./run.sh

Query Historical Data

from src.storage import StorageManager

storage = StorageManager()

# Get last 24 hours for AAPL
data = storage.get_historical_data("AAPL", hours=24, limit=1000)

# Get statistics
stats = storage.get_statistics("AAPL")
print(f"Min: ${stats['min_price']:.2f}")
print(f"Max: ${stats['max_price']:.2f}")

Custom Analytics

from src.analytics import StockAnalytics

analytics = StockAnalytics()

# Calculate indicators
indicators = analytics.calculate_indicators("AAPL", stock_data)

# Get trading signals
signals = analytics.get_trading_signals("AAPL", indicators)
print(signals)  # {'rsi_signal': 'OVERSOLD', 'ma_signal': 'BULLISH'}

🧪 Testing

Run unit tests:

source venv/bin/activate
pytest tests/ -v --cov=src

Run specific test:

pytest tests/test_analytics.py -v

📊 Performance Metrics

• Latency: < 100ms end-to-end
• Throughput: 5+ stocks @ 2-second intervals = 150+ messages/minute
• Memory Usage: ~200MB (with 5 stocks, 1000 data points each)
• Storage: ~10MB per day (5 stocks)

🐛 Troubleshooting

Redis Connection Failed

# Check if Redis is running
redis-cli ping

# If not running, start it
redis-server --daemonize yes

No Data in Dashboard

1. Check producer is running: ps aux | grep producer
2. Check consumer is running: ps aux | grep consumer
3. Check Redis stream: redis-cli XLEN stock_prices
4. Check logs: tail -f logs/app.log

Import Errors

# Ensure virtual environment is activated
source venv/bin/activate

# Reinstall dependencies
pip install -r requirements.txt

Port Already in Use

# Kill process on port 8501 (Streamlit)
lsof -ti:8501 | xargs kill -9

# Or use different port
streamlit run dashboard/app.py --server.port 8502

🚀 Deployment Options

Local Docker

docker-compose up -d

Streamlit Cloud

1. Push to GitHub
2. Go to share.streamlit.io
3. Deploy dashboard/app.py
4. Configure secrets for Redis connection

AWS EC2

# Install dependencies on EC2
./setup.sh

# Run with nohup
nohup ./run.sh &

# Access via EC2 public IP:8501

📝 Future Enhancements

• Add authentication and user management
• Implement price alerts via email/SMS
• Add backtesting capabilities
• Support for cryptocurrency data
• Machine learning price predictions
• Multi-timeframe analysis
• Portfolio tracking and management
• Export reports to PDF
• REST API endpoints
• WebSocket for real-time updates

🤝 Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

👤 Author

Ravi Shankar

• GitHub: @ravishankarjs
• LinkedIn: @Ravi Shankar J S

🙏 Acknowledgments

• yfinance for stock market data
• Streamlit for the dashboard framework
• Redis for streaming capabilities
• Plotly for interactive charts

📸 Screenshots

Dashboard Overview

Price Chart with Indicators

Technical Analysis Panel

---

⭐ If you find this project helpful, please give it a star! ⭐

💼 Perfect for showcasing in your portfolio and interviews!