Skip to content
/ FieldStream Public

FieldStream is a secure data streaming system designed for researchers and field devices. It safely bridges the gap between remote IoT sensors and corporate networks using WebSocket and MQTT protocols with Zero Trust security principles, end-to-end encryption, and automated certificate management.

0 stars 0 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

tazomatalax/FieldStream

BranchesTags

Repository files navigation

FieldStream

Secure Field-to-Corporate Data Streaming

FieldStream is a secure data streaming system designed for researchers and field devices. It safely bridges the gap between remote IoT sensors and corporate networks using WebSocket and MQTT protocols with Zero Trust security principles, end-to-end encryption, and automated certificate management.

🎯 Why FieldStream?

For Researchers:

  • ✅ 15-minute setup with pre-configured bundles
  • ✅ Works anywhere - WiFi, cellular, or ethernet
  • ✅ Handles all data types - sensors, images, events automatically
  • ✅ Built-in diagnostics - know exactly what's happening
  • ✅ Offline resilience - never lose data due to connectivity issues

For IT Departments:

  • 🔒 Enterprise security - Zero Trust, mTLS, micro-segmentation
  • 🏗️ Corporate integration - Direct to InfluxDB, existing infrastructure
  • 📈 Scalable architecture - Handles thousands of devices
  • 🔧 Easy management - Automated certificates, device bundles
  • 📊 Full visibility - Comprehensive monitoring and logging

🏗️ System Architecture

This system implements a three-tier security architecture designed for deployment across multiple machines:

graph TB
    subgraph "Field Devices (Remote Locations)"
        FC1[Field Client 1]
        FC2[Field Client 2]
        FCN[Field Client N]
    end
    
    subgraph "DMZ Server (Public Cloud/VPS)"
        subgraph "DMZ Network Segments"
            subgraph "Web Tier (172.20.1.0/24)"
                CADDY[Caddy Reverse Proxy<br/>:80/:443]
                WS[WebSocket Server<br/>:8080]
            end
            
            subgraph "MQTT Tier (172.20.2.0/24)"
                DMZMQ[DMZ MQTT Broker<br/>:8883 TLS]
            end
            
            subgraph "Bridge Tier (172.20.3.0/24)"
                BRIDGE[Internal Bridge<br/>mTLS]
            end
        end
    end
    
    subgraph "Internal Network (Private Infrastructure)"
        subgraph "Internal Services"
            INTMQ[Internal MQTT Broker<br/>:8883 mTLS]
            DIST[Data Distributor]
            INFLUX[InfluxDB]
            STORAGE[File Storage<br/>Images/Documents]
        end
    end
    
    subgraph "Internet"
        NET[Internet<br/>TLS 1.3 Encrypted]
    end
    
    FC1 -.->|WSS Connection| NET
    FC2 -.->|WSS Connection| NET
    FCN -.->|WSS Connection| NET
    
    NET -->|Port 443/80| CADDY
    CADDY -->|Proxy| WS
    WS -->|mTLS| DMZMQ
    DMZMQ -->|Bridge mTLS| BRIDGE
    BRIDGE -.->|mTLS Tunnel| INTMQ
    INTMQ -->|Subscribe| DIST
    DIST -->|Time Series| INFLUX
    DIST -->|Files| STORAGE
    
    classDef fieldDevice fill:#e1f5fe
    classDef dmzComponent fill:#fff3e0  
    classDef internalComponent fill:#e8f5e8
    classDef storage fill:#fce4ec
    
    class FC1,FC2,FCN fieldDevice
    class CADDY,WS,DMZMQ,BRIDGE dmzComponent
    class INTMQ,DIST internalComponent
    class INFLUX,STORAGE storage
Loading

📊 Data Flow Architecture

sequenceDiagram
    participant FC as Field Client
    participant C as Caddy Proxy
    participant WS as WebSocket Server
    participant DMQ as DMZ MQTT
    participant IMQ as Internal MQTT
    participant DD as Data Distributor
    participant IDB as InfluxDB
    participant FS as File Storage
    
    Note over FC,FS: Secure Data Pipeline
    
    FC->>+C: WSS Connection + Client Cert
    C->>+WS: Proxy WebSocket
    WS->>+DMQ: Publish (mTLS)
    DMQ->>+IMQ: Bridge (mTLS Tunnel)
    IMQ->>+DD: Subscribe to Topics
    
    alt Time Series Data
        DD->>+IDB: Write Metrics
        IDB-->>-DD: ACK
    else File Data
        DD->>+FS: Store File
        FS-->>-DD: ACK
    else Command Data
        DD->>+IMQ: Publish Command
        IMQ->>+DMQ: Bridge Response
        DMQ->>+WS: Forward Command
        WS->>+C: WebSocket Message
        C-->>-FC: Command Response
    end
    
    DD-->>-IMQ: Processing Complete
    IMQ-->>-DMQ: Bridge ACK
    DMQ-->>-WS: Publish ACK
    WS-->>-C: Success Response
    C-->>-FC: ACK
Loading

🌐 Multi-Machine Deployment Architecture

This system is designed for deployment across three separate machines:

Machine 1: Field Devices (Edge)

  • Location: Remote field locations, industrial sites, IoT deployments
  • Components: Field clients only
  • Network: Outbound internet access required
  • Security: Client certificates, device attestation

Machine 2: DMZ Server (Public)

  • Location: Public cloud (AWS, GCP, Azure) or VPS with public IP
  • Components: Caddy, WebSocket Server, DMZ MQTT Broker
  • Network: Public internet exposure (ports 80/443)
  • Security: Let's Encrypt certificates, firewall rules, micro-segmentation

Machine 3: Internal Network (Private)

  • Location: Corporate network, private cloud, on-premises
  • Components: Internal MQTT Broker, Data Distributor, InfluxDB integration
  • Network: Private network, VPN/tunnel to DMZ
  • Security: Internal certificates, network isolation

📋 Prerequisites

All Machines

  • Docker & Docker Compose
  • OpenSSL (for certificate generation)

DMZ Server

  • Public IP address
  • Domain name pointing to the server
  • Ports 80/443 accessible from internet

Internal Network

  • InfluxDB instance
  • Network connectivity to DMZ server (VPN/tunnel)
  • File storage system for non-time-series data

🚀 Deployment Guide

Step 1: Certificate Generation (Run Once)

On any machine with this repository:

# Clone repository
git clone https://github.com/tazomatalax/FieldStream
cd data-websocket

# Generate certificates for your domain
chmod +x scripts/generate-certificates.sh
./scripts/generate-certificates.sh your-dmz-domain.com

# This creates a 'certs/' directory with all required certificates

Step 2: DMZ Server Deployment

On your public DMZ server:

# Transfer repository and certificates to DMZ server
scp -r data-websocket user@dmz-server:/opt/

# On DMZ server
cd /opt/data-websocket

# Configure environment
cp .env.example .env
nano .env
# Set DOMAIN_NAME=your-actual-domain.com
# Set ADMIN_EMAIL=your-email@domain.com

# Deploy DMZ services
cd dmz/
docker-compose up --build -d

# Verify deployment
docker-compose logs -f

Step 3: Internal Network Deployment

On your internal network server:

# Transfer repository and certificates to internal server
scp -r data-websocket user@internal-server:/opt/

# On internal server
cd /opt/data-websocket

# Configure environment for internal network
cp .env.example .env.internal
nano .env.internal
# Configure InfluxDB settings (see configuration section below)

# Deploy internal services
cd internal-network/
docker-compose -f docker-compose.yml --env-file ../.env.internal up --build -d

# Verify deployment
docker-compose logs -f

Step 4: Field Client Deployment

On each field device:

# Transfer client components and certificates
scp -r data-websocket/field-client user@field-device:/opt/
scp -r data-websocket/certs user@field-device:/opt/field-client/

# On field device
cd /opt/field-client

# Configure unique device ID
export DEVICE_ID="field-device-$(hostname)-$(date +%s)"
export DOMAIN_NAME="your-dmz-domain.com"

# Deploy field client
docker-compose up --build -d

# Verify connection
docker-compose logs -f

⚙️ Configuration

InfluxDB Integration

Environment Variables (.env.internal):

# InfluxDB Configuration (Version 2.x)
INFLUXDB_URL=http://your-influx-server:8086
INFLUXDB_TOKEN=your-influx-api-token
INFLUXDB_ORG=your-organization
INFLUXDB_BUCKET=sensor-data

# InfluxDB v1.x (if using legacy version)
INFLUXDB_URL=http://your-influx-server:8086
INFLUXDB_DATABASE=sensor_data
INFLUXDB_USERNAME=your-username
INFLUXDB_PASSWORD=your-password

# File Storage
FILE_STORAGE_PATH=/data/files
FILE_STORAGE_TYPE=local  # Options: local, s3, azure, gcs

Supported Data Types

The system automatically handles all data types:

// Time-series data (sent to InfluxDB)
{
  "deviceId": "sensor-001",
  "dataType": "timeseries",
  "payload": {
    "temperature": 23.5,
    "humidity": 65.2,
    "pressure": 1013.25,
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

// File data (images, documents)
{
  "deviceId": "camera-001", 
  "dataType": "file",
  "payload": {
    "filename": "image_20240101_120000.jpg",
    "contentType": "image/jpeg",
    "data": "base64-encoded-content",
    "metadata": {
      "size": 1024000,
      "timestamp": "2024-01-01T12:00:00Z"
    }
  }
}

// Event data
{
  "deviceId": "alarm-001",
  "dataType": "event", 
  "payload": {
    "eventType": "alarm",
    "severity": "high",
    "message": "Temperature threshold exceeded",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

// Command responses
{
  "deviceId": "actuator-001",
  "dataType": "response",
  "payload": {
    "commandId": "cmd-12345",
    "status": "completed",
    "result": "valve opened",
    "timestamp": "2024-01-01T12:00:00Z"
  }
}

🛠️ Operations

Monitoring

# Check DMZ services
docker-compose -f dmz/docker-compose.yml logs -f

# Check internal services  
docker-compose -f internal-network/docker-compose.yml logs -f

# Check field clients
docker-compose -f field-client/docker-compose.yml logs -f

Scaling

Adding new field devices:

# Generate new client certificate
./scripts/generate-client-cert.sh new-device-id

# Deploy to new device with unique DEVICE_ID

Adding DMZ capacity:

# Scale WebSocket servers
docker-compose -f dmz/docker-compose.yml up --scale websocket-server=3

🔧 Network Security

Firewall Configuration

DMZ Server iptables rules:

# Run the firewall setup script
sudo ./scripts/setup-firewall.sh

Internal Network:

# Allow only DMZ bridge connection
iptables -A INPUT -s DMZ_SERVER_IP -p tcp --dport 18883 -j ACCEPT
iptables -A INPUT -p tcp --dport 18883 -j DROP

📊 Data Pipeline Performance

  • Throughput: 10,000+ messages/second per WebSocket server
  • Latency: <100ms end-to-end (field device to InfluxDB)
  • Reliability: 99.9% message delivery with MQTT QoS 1
  • Scalability: Horizontal scaling of all components

🆘 Troubleshooting

Common Issues

Connection refused from field device:

  • Verify domain name resolution
  • Check DMZ server firewall (ports 80/443)
  • Validate client certificates

MQTT bridge not connecting:

  • Check network connectivity between DMZ and internal
  • Verify certificate permissions
  • Review MQTT broker logs

Data not reaching InfluxDB:

  • Check InfluxDB credentials in .env.internal
  • Verify data distributor logs
  • Test InfluxDB connectivity

Debug Commands

# Test WebSocket connection
wscat -c wss://your-domain.com/ws

# Test MQTT connectivity  
mosquitto_pub -h dmz-server -p 8883 --cafile certs/ca.crt --cert certs/client.crt --key certs/client.key -t test -m "hello"

# Check certificate validity
openssl x509 -in certs/client.crt -text -noout

About

FieldStream is a secure data streaming system designed for researchers and field devices. It safely bridges the gap between remote IoT sensors and corporate networks using WebSocket and MQTT protocols with Zero Trust security principles, end-to-end encryption, and automated certificate management.

Topics

mqtt iot websockets zero-trust

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Languages