CoherenceLab

Distributed Entropy Collection and Coherence Analysis Platform

CoherenceLab is a multi-service research platform designed to collect random entropy samples from distributed devices and analyze them for patterns of randomness and potential coherence phenomena.

---

🎯 Overview

CoherenceLab enables researchers to:

1. Collect entropy samples from multiple distributed devices (IoT, mobile, desktop)
2. Store samples in a PostgreSQL database with automatic rolling-window management
3. Archive older data to Storj decentralized storage
4. Analyze collected entropy using statistical randomness tests
5. Visualize coherence patterns across time and space via a web dashboard

---

🏗️ Architecture

📚 For detailed architecture documentation, see docs/architecture.md

The architecture document includes:

• System overview with component diagrams
• End-to-end data flow sequences
• Data retention lifecycle (Hot → Warm → Cold)
• Module responsibilities and interactions
• Database schema with relationships
• Performance considerations and scalability

High-Level System Overview

flowchart LR
    subgraph Devices["IoT Devices"]
        D1[Device 1]
        D2[Device 2]
        D3[Device N]
    end

    subgraph Backend["Backend Collector<br/>(Node.js + Fastify)"]
        WS[WebSocket]
        API[REST API]
        Stats[Statistics]
    end

    subgraph Storage["Data Storage"]
        PG[(PostgreSQL)]
        Storj[Storj Network<br/>4GB Chunks]
    end

    subgraph Analyzer["Analyzer<br/>(Python + FastAPI)"]
        Tests[Statistical<br/>Tests]
    end

    subgraph UI["Web UI<br/>(React + Vite)"]
        Dash[Dashboard]
        Metrics[Metrics]
    end

    Devices -->|Entropy| WS
    UI <-->|HTTP| API
    WS --> PG
    Stats --> PG
    Stats --> Storj
    PG <--> Tests
    Storj <-.-> Tests
    API --> Metrics

    style Devices fill:#e1f5ff
    style Backend fill:#fff4e1
    style Storage fill:#f0e1ff
    style Analyzer fill:#e1ffe1
    style UI fill:#ffe1e1

Loading

Core Components

Component	Technology	Key Features
Backend Collector	Node.js + Fastify + TypeScript	JWT auth, WebSocket ingestion, rolling window, Storj archival, statistics computation
Analyzer Service	Python + FastAPI	NIST tests, autocorrelation, cross-correlation, job queue, re-analysis from archives
Web UI	React + Vite + TypeScript	Device management, real-time dashboard, long-term metrics, analysis explorer
PostgreSQL	PostgreSQL 16+	Hot data (~150GB rolling), long-term statistics (indefinite), chunk catalog
Storj Network	Decentralized Storage	4GB chunks, .meta.json sidecars, disaster recovery, re-analysis capability

---

🚀 Quick Start

Prerequisites

• Docker and Docker Compose (recommended)
• OR:
  • Node.js 20+
  • Python 3.11+
  • PostgreSQL 16+

Option 1: Docker (Recommended)

1. Clone the repository

   git clone https://github.com/yourusername/CoherenceLab.git
   cd CoherenceLab

2. Set up environment variables

   cp .env.example .env
   # Edit .env with your settings

3. Start all services

   docker-compose up -d

4. Access the application

   • Web UI: http://localhost
   • Backend API: http://localhost:3000
   • Analyzer API: http://localhost:8000

5. Create your first user

   • Navigate to http://localhost/signup
   • Create an account
   • Add a device and get your device token

Option 2: Local Development

1. Database Setup

# Start PostgreSQL (via Docker for development)
docker-compose -f docker-compose.dev.yml up -d

2. Backend Collector

cd apps/backend-collector

# Install dependencies
npm install

# Generate Prisma client
npx prisma generate

# Run migrations
npx prisma migrate dev

# Start development server
npm run dev

3. Analyzer Service

cd apps/analyzer

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Start development server
uvicorn src.main:app --reload --port 8000

4. Web Frontend

cd apps/web

# Install dependencies
npm install

# Start development server
npm run dev

---

📝 Configuration

Environment Variables

Key configuration variables (see .env.example for full list):

Database:

DATABASE_URL=postgresql://coherence:coherence@localhost:5432/coherencelab

Backend:

PORT=3000
JWT_SECRET=your-super-secret-jwt-key
CORS_ORIGIN=http://localhost:5173

Storj (Optional):

STORJ_ACCESS_GRANT=your-storj-access-grant
STORJ_BUCKET=coherencelab-entropy
STORJ_CHUNK_SIZE_GB=4

Rolling Window:

MAX_DB_SIZE_GB=150
MIN_RETENTION_DAYS=7
MAINTENANCE_INTERVAL_HOURS=1

Analyzer:

ANALYZER_API_KEY=your-analyzer-api-key

---

🔧 Usage Guide

1. Register and Create a Device

1. Sign up at the web UI
2. Navigate to Devices → Add Device
3. Enter device name and type
4. Copy the device token (you'll need this for client setup)

2. Configure Sampling Profile

1. Open your device detail page
2. Create a sampling profile:
   • Sample Rate: e.g., 1.0 Hz (1 sample per second)
   • Sample Size: e.g., 1024 bytes
   • Label: "Primary entropy source"

3. Connect Your Entropy Source

Example WebSocket client (JavaScript):

const WebSocket = require('ws');
const crypto = require('crypto');

const DEVICE_TOKEN = 'your-device-token-here';
const PROFILE_ID = 'your-profile-id-here';

const ws = new WebSocket(`ws://localhost:3000/ws/entropy?token=${DEVICE_TOKEN}`);

ws.on('open', () => {
  console.log('Connected to CoherenceLab');
});

ws.on('message', (data) => {
  const message = JSON.parse(data);
  console.log('Server:', message);

  if (message.type === 'auth_success') {
    // Start sending entropy samples
    setInterval(() => {
      // Generate random bytes (replace with your actual entropy source)
      const entropyBytes = crypto.randomBytes(1024);

      ws.send(JSON.stringify({
        timestamp: new Date().toISOString(),
        sequence: Date.now(),
        profileId: PROFILE_ID,
        bytes: entropyBytes.toString('base64'),
      }));
    }, 1000); // Send every second
  }
});

4. Run Analysis

1. Navigate to Analysis → New Analysis
2. Select date range
3. Choose tests:
   • Frequency Test: Basic randomness check
   • Entropy Test: Shannon entropy
   • Runs Test: Bit sequence patterns
   • Autocorrelation: Temporal patterns
   • Cross-Correlation: Spatial coherence between devices
4. Click Create Job
5. View results on the job detail page

---

📊 Statistical Tests

Frequency (Monobit) Test

Tests whether the proportion of 1s and 0s is approximately equal (NIST SP 800-22).

• Pass criteria: p-value ≥ 0.01

Entropy Test

Measures Shannon entropy to assess randomness.

• Pass criteria: Entropy ratio > 0.95

Runs Test

Checks for expected number of uninterrupted sequences of identical bits.

• Pass criteria: p-value ≥ 0.01

Autocorrelation Test

Detects patterns or periodicities within a single device's stream.

• Pass criteria: Max absolute correlation < 0.1

Cross-Correlation Test

Measures correlation between different devices over shared time windows.

• Pass criteria: Max absolute correlation < 0.1
• Research Interest: Significant correlation may indicate coherence phenomena!

---

🗄️ Database Management

Rolling Window

CoherenceLab automatically maintains a ~150GB rolling window:

1. Hourly maintenance job checks database size
2. When size exceeds 150GB:
   • Oldest archived samples (linked to Storj) are deleted
   • Only samples older than 7 days are eligible
3. VACUUM runs to reclaim disk space

Storj Archival

1. Samples are appended to a local spool file as they arrive
2. When spool reaches 4GB, it's uploaded to Storj
3. Database records are updated with storj_chunk_id
4. Local spool file is kept for 24h then deleted

To retrieve archived data:

-- Find Storj chunks for a time range
SELECT * FROM storj_chunks
WHERE min_timestamp >= '2025-01-01'
  AND max_timestamp <= '2025-01-31';

---

🐳 Docker Commands

# Start all services
docker-compose up -d

# View logs
docker-compose logs -f

# Stop all services
docker-compose down

# Rebuild after code changes
docker-compose up -d --build

# Run database migrations
docker-compose exec backend-collector npx prisma migrate deploy

# Access PostgreSQL
docker-compose exec postgres psql -U coherence -d coherencelab

---

🧪 Development

Project Structure

CoherenceLab/
├── apps/
│   ├── backend-collector/    # Node.js backend
│   ├── analyzer/              # Python analyzer
│   └── web/                   # React frontend
├── packages/
│   └── shared/                # Shared types
├── docker-compose.yml
└── README.md

Running Tests

# Backend (TODO: Add tests)
cd apps/backend-collector
npm run test

# Analyzer (TODO: Add tests)
cd apps/analyzer
pytest

# Frontend (TODO: Add tests)
cd apps/web
npm run test

Database Migrations

cd apps/backend-collector

# Create new migration
npx prisma migrate dev --name your_migration_name

# Apply migrations in production
npx prisma migrate deploy

# Open Prisma Studio (GUI)
npx prisma studio

---

🔐 Security Considerations

• Device Tokens: Treat as secrets. Rotate if compromised.
• JWT Secret: Use a strong random secret in production.
• HTTPS: Use reverse proxy (nginx, Caddy) with SSL/TLS.
• API Keys: Change default analyzer API key.
• Database: Restrict PostgreSQL network access.
• Storj: Keep access grants secure and scoped.

---

📈 Scaling Considerations

For high-volume deployments:

1. Horizontal Scaling:

   • Run multiple backend collector instances behind a load balancer
   • Use sticky sessions for WebSocket connections

2. Database Optimization:

   • Enable PostgreSQL partitioning on entropy_samples by timestamp
   • Use read replicas for analyzer queries

3. Caching:

   • Add Redis for session management and hot data

4. Queue System:

   • Replace in-memory jobs with RabbitMQ or Redis Queue
   • Run analyzer workers in separate containers

5. Monitoring:

   • Add Prometheus metrics
   • Set up Grafana dashboards
   • Configure alerting (PagerDuty, Slack)

---

🛣️ Roadmap

MVP (Current)

• User authentication
• Device management
• WebSocket entropy ingestion
• PostgreSQL storage with rolling window
• Storj integration (skeleton)
• Basic randomness tests
• Cross-correlation analysis
• Web dashboard

Future Enhancements

• Real-time charts and visualizations
• Geographic heatmaps for spatial coherence
• Advanced statistical tests (FFT, spectral analysis)
• Export analysis results (CSV, PDF)
• Mobile apps (iOS, Android)
• Multi-user collaboration features
• API rate limiting
• Comprehensive test coverage
• Performance benchmarking
• Admin dashboard

---

📜 License

AGPLv3 License - See LICENSE file for details

---

🤝 Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

---

📧 Contact

For questions, issues, or research collaboration:

• GitHub Issues: https://github.com/yourusername/CoherenceLab/issues

---

🙏 Acknowledgments

• NIST SP 800-22 for randomness testing frameworks
• Storj for decentralized storage
• The open-source community

---

Happy researching! 🔬