🔢 Math Constants Storage System

⚠️ IMPORTANT: DATA FILES NOT INCLUDED

This repository contains the API and storage infrastructure only. See Data Sources section for obtaining data files.

High-performance, triple-redundancy storage system for accessing billions of digits of mathematical constants (π, φ, e, √2, and more) with FastAPI backend, Redis caching, WebSocket streaming, dedicated endpoints per constant, and accuracy-first design.

✨ Features

• 12 Mathematical Constants: Pi, Euler's number, Golden ratio, Square roots, Logarithms, and more
• Dedicated Endpoints: Each constant has its own clean API namespace (/pi/*, /e/*, etc.)
• Triple Redundancy: Original file + SQLite chunks + Binary cache
• Redis Caching: Lightning-fast access with intelligent cache invalidation
• WebSocket Streaming: Real-time digit streaming with chunked delivery
• Smart Cache Management: Automatically skips already-built caches
• Accuracy First: Automatic verification and corruption detection
• High Performance: Optimized for random access to billions of digits
• Modular Architecture: Clean separation with 16 focused router files
• Hot Reloading: Development-friendly Docker setup
• Backward Compatible: Legacy parameterized endpoints still work

🚀 Quick Start

# Clone the repository
git clone https://github.com/pkeffect/numbers
cd numbers

# Copy environment configuration
cp .env.example .env

# Place your math constant files in data/ directory
# Required filenames: pi_digits.txt, e_digits.txt, phi_digits.txt, etc.

# Start the development environment (includes Redis)
docker-compose up --build

# Build caches (after first startup)
curl -X POST http://localhost:8000/admin/build-all-caches

🆕 New Features: Redis & WebSocket

Redis Caching

Benefits:

• ⚡ 10-100x faster response times for cached queries
• 📊 Automatic cache hit/miss tracking
• 🔄 Smart cache invalidation on data updates
• 💾 Configurable TTL and memory limits

Cached Operations:

• Digit retrieval
• Search results
• Statistical analysis

Cache Control:

# Use cache (default)
GET /pi/digits?start=0&length=100&use_cache=true

# Bypass cache
GET /pi/digits?start=0&length=100&use_cache=false

# Clear cache for a constant
DELETE /pi/cache

# Get cache statistics
GET /health  # includes cache stats

WebSocket Streaming

Real-time Features:

• 📡 Live digit streaming
• 🔔 Cache build notifications
• 📊 Live status updates
• 🎯 Per-constant subscriptions

WebSocket Endpoints:

// General connection (system-wide events)
ws://localhost:8000/ws/connect

// Constant-specific connection
ws://localhost:8000/ws/constant/pi

// Simplified streaming
ws://localhost:8000/ws/stream/pi?start=0&length=1000

Example Usage:

const ws = new WebSocket('ws://localhost:8000/ws/constant/pi');

ws.onopen = () => {
    // Stream 1000 digits in chunks of 100
    ws.send(JSON.stringify({
        command: 'stream_digits',
        start: 0,
        length: 1000,
        chunk_size: 100
    }));
};

ws.onmessage = (event) => {
    const data = JSON.parse(event.data);
    
    if (data.type === 'stream_chunk') {
        console.log(`Chunk ${data.chunk_number}: ${data.digits}`);
        console.log(`Remaining: ${data.remaining} digits`);
    }
};

Test WebSocket: Open http://localhost:3000/websocket_test.html in your browser for an interactive test interface.

🔧 API Endpoints

General Endpoints

GET  /                    # API root with system info
GET  /health              # System health check (includes Redis stats)
GET  /constants           # List all constants with status

Per-Constant Endpoints (Pattern for all 12 constants)

Each constant has 8 dedicated endpoints (7 original + 1 new):

GET    /{constant}/status        # Status and cache information
GET    /{constant}/digits        # Retrieve digits (Redis cached)
GET    /{constant}/search        # Search sequences (Redis cached)
GET    /{constant}/stats         # Statistical analysis (Redis cached)
GET    /{constant}/random        # Get random digits
POST   /{constant}/build-cache   # Build SQLite and binary cache
POST   /{constant}/verify        # Verify data integrity
DELETE /{constant}/cache         # Clear Redis cache (NEW!)

WebSocket Endpoints

WS /ws/connect                    # General system connection
WS /ws/constant/{constant_id}     # Constant-specific connection
WS /ws/stream/{constant_id}       # Simplified streaming

Admin Endpoints

POST /admin/build-all-caches   # Build caches for all constants
GET  /admin/status             # Administrative status

🐳 Docker Services

The system now includes Redis:

# Start all services (API + Redis)
docker-compose up

# Start with debug tools (API + Redis + Redis Commander + SQLite Browser)
docker-compose --profile debug up

# Start with frontend (API + Redis + Nginx frontend)
docker-compose --profile frontend up

Services:

• math-constants-api (port 8000) - Main API
• redis (port 6379) - Caching layer
• redis-commander (port 8081, debug profile) - Redis web UI
• sqlite-browser (port 8080, debug profile) - SQLite web UI
• frontend (port 3000, frontend profile) - Test interface

📊 Performance Improvements

Before Redis (Direct SQLite/File):

• Average response time: 50-100ms
• Search queries: 200-500ms
• Stats calculations: 300-600ms

After Redis Caching:

• Cached digit retrieval: 5-10ms (90% faster)
• Cached search results: 2-5ms (95% faster)
• Cached statistics: 3-8ms (95% faster)

Cache Hit Rates:

• Digit queries: ~60-80%
• Search queries: ~40-60%
• Stats queries: ~70-90%

🔄 Cache Invalidation

Redis cache is automatically invalidated when:

• Cache is rebuilt via /admin/build-all-caches
• Cache is rebuilt for specific constant via /{constant}/build-cache
• Manual cache clear via DELETE /{constant}/cache

📡 WebSocket Commands

General Connection Commands:

// Ping-pong for keepalive
{ "type": "ping", "timestamp": 1234567890 }

// Get connection statistics
{ "type": "get_stats" }

Constant-Specific Commands:

// Stream digits
{
    "command": "stream_digits",
    "start": 0,
    "length": 1000,
    "chunk_size": 100
}

// Get constant status
{ "command": "get_status" }

// Ping
{ "command": "ping" }

Server Messages:

// Stream chunk
{
    "type": "stream_chunk",
    "chunk_number": 5,
    "position": 500,
    "digits": "1234567890...",
    "remaining": 500
}

// Broadcast notifications
{
    "type": "digits_accessed",
    "constant_id": "pi",
    "start": 100,
    "length": 50
}

{
    "type": "cache_build_complete",
    "constant_id": "pi"
}

⚙️ Configuration

Key environment variables for Redis & WebSocket (see .env.example):

# Redis Configuration
REDIS_URL=redis://redis:6379/0
REDIS_MAX_CONNECTIONS=50
CACHE_TTL=3600
ENABLE_CACHE=true

# WebSocket Configuration
WS_HEARTBEAT_INTERVAL=30
WS_MAX_CONNECTIONS=1000
WS_MESSAGE_QUEUE_SIZE=100

🧪 Testing

Test Redis Connection:

# Connect to Redis CLI
docker-compose exec redis redis-cli

# Check keys
KEYS mathconst:*

# Get cache stats
INFO stats

Test WebSocket:

# Using wscat (install: npm install -g wscat)
wscat -c ws://localhost:8000/ws/connect

# Send ping
{"type":"ping","timestamp":1234567890}

Test Caching:

# First request (cache miss)
time curl "http://localhost:8000/pi/digits?start=0&length=100"

# Second request (cache hit - much faster!)
time curl "http://localhost:8000/pi/digits?start=0&length=100"

🤝 Contributing

See CONTRIBUTING.md for detailed guidelines.

📄 License

MIT License - see LICENSE.md

🔮 Roadmap

• Redis caching layer
• WebSocket streaming support
• Redis pub/sub for distributed deployments
• GraphQL API support
• Advanced pattern recognition with ML
• Real-time collaborative features
• Kubernetes deployment manifests

📞 Support

• 📚 Full Documentation
• 🐛 Issue Tracker
• 💬 Discussions

---

New in v2.1: Redis caching and WebSocket streaming for real-time applications!