D-GRID: Decentralized Git-Relay Infrastructure for Distributed Tasks

🚀 Blitz-Launch Operation - Use Git as a transactional and distributed state database for decentralized task execution.

🎯 Vision

D-GRID is a serverless, decentralized task execution system where:

• 🗂️ Git as Database - All system state (nodes, tasks, results) is tracked on GitHub as the single source of truth
• 📊 Auto-Generated Dashboard - Public, real-time status dashboard on GitHub Pages
• 🤖 Autonomous Worker Nodes - Python workers in Docker containers that self-register, execute tasks, and report results
• ✅ Zero Central Dependencies - No central APIs, no external databases, no orchestrators needed

🏗️ Architecture

┌─────────────────────────────────────────────────────────┐
│                  GitHub Repository                      │
│  (Single Source of Truth - Distributed State)          │
├─────────────────────────────────────────────────────────┤
│  nodes/                  → Active node registry         │
│  tasks/queue/            → Pending tasks               │
│  tasks/in_progress/      → Running tasks (atomic)      │
│  tasks/completed/        → Successful tasks            │
│  tasks/failed/           → Failed tasks                │
└─────────────────────────────────────────────────────────┘
         ↕ (Pull/Push via Git)
┌─────────────────────────────────────────────────────────┐
│         GitHub Pages (Status Dashboard)                │
│  (Read-only, auto-updated by GitHub Actions)          │
└─────────────────────────────────────────────────────────┘
         ↑ (Generated by)
┌─────────────────────────────────────────────────────────┐
│         GitHub Actions (Orchestration)                 │
│  • Task submission validation                          │
│  • Dashboard generation & cleanup                      │
└─────────────────────────────────────────────────────────┘
         ↑ (Triggered by)
┌──────────┬──────────┬──────────┬──────────┐
│ Worker 1 │ Worker 2 │ Worker 3 │ Worker N │
│ (Python) │ (Python) │ (Python) │ (Python) │
│ (Docker) │ (Docker) │ (Docker) │ (Docker) │
└──────────┴──────────┴──────────┴──────────┘

🔧 Technology Stack

Component	Technology	Why
Worker Node	Python 3.11	Fast development, mature libraries, great community
Containerization	Docker Alpine	Lightweight (~150MB), secure, includes Python
Coordination	Git + GitHub	Single source of truth, decentralized, built-in history
Dashboard	GitHub Pages + HTML	Free, integrated, no extra infrastructure
CI/CD	GitHub Actions	Free, serverless, GitHub-native

🚀 How It Works

1. Worker Node Lifecycle

┌─────────────────────────────────────────────┐
│ Worker starts (docker run)                  │
└──────────────┬──────────────────────────────┘
               │
               ↓
┌─────────────────────────────────────────────┐
│ 1. Clone repo (or pull if exists)          │
│ 2. Self-register: nodes/{id}.json          │
│ 3. Commit and push registration            │
└──────────────┬──────────────────────────────┘
               │
               ↓
        ┌──────────────┐
        │ LOOP START   │
        └──────────────┘
               │
        ┌──────┴──────┐
        │ Pull rebase │  (Always up-to-date)
        └──────┬──────┘
               │
        ┌──────┴────────────────────┐
        │ Task available?           │
        └──────┬─────────────────────┘
               │
        ┌──────┴──────────────────────────────┐
        │                                     │
        │                                     │
    [YES]│                                    │[NO]
        │                                     │
        ↓                                     ↓
┌─────────────────┐              ┌──────────────────┐
│ git mv atomic   │              │ Send heartbeat   │
│ queue→in_prog   │              │ Update node      │
│ (transaction!)  │              │ .last_heartbeat  │
└────────┬────────┘              └────────┬─────────┘
         │                                │
         ↓                                │
┌─────────────────┐                       │
│ Commit and push │                       │
│ (First to push  │                       │
│  wins!)         │                       │
└────────┬────────┘                       │
         │                                │
         ↓                                │
┌─────────────────┐                       │
│ Execute Docker  │                       │
│ container       │                       │
└────────┬────────┘                       │
         │                                │
         ↓                                │
┌─────────────────┐                       │
│ Report result   │                       │
│ completed/ or   │                       │
│ failed/         │                       │
└────────┬────────┘                       │
         │                                │
         │                    ┌───────────┘
         ↓                    ↓
        ┌──────────────────────────┐
        │ Sleep (PULL_INTERVAL)    │
        └──────────────┬───────────┘
                       │
                       └─ (back to Pull)

2. Task Submission Flow

Fork → Create task in tasks/queue/ → Pull Request →
Merge (auto-validated) → GitHub Actions scans state →
Worker node finds it → Execution → Result pushed back

📋 Directory Structure

d-grid/
├── .github/
│   └── workflows/
│       ├── process-task-pr.yml     # Validate task submissions
│       └── update-gh-pages.yml     # Generate dashboard & cleanup
├── nodes/
│   └── {node_id}.json              # State of each node
├── tasks/
│   ├── queue/                      # Pending tasks
│   ├── in_progress/                # Running tasks (atomic acquisition)
│   ├── completed/                  # Successfully completed tasks
│   └── failed/                     # Failed tasks
├── worker/
│   ├── main.py                     # Worker entry point
│   ├── config.py                   # Configuration & validation
│   ├── logger_config.py            # Unified logging
│   ├── git_handler.py              # Git operations
│   ├── state_manager.py            # Node registration & heartbeat
│   ├── task_runner.py              # Task execution
│   ├── web_server.py               # Local worker dashboard
│   └── requirements.txt            # Python dependencies
├── docs/
│   └── index.html                  # GitHub Pages dashboard
├── .gitignore
├── Dockerfile                       # Worker image build
├── docker-compose.yml              # Local development setup
├── README.md                        # This file
└── progress.md                      # Development progress tracking

🎲 Task Format

Submit tasks as JSON files in tasks/queue/:

{
  "task_id": "task-001",
  "script": "echo 'Hello, D-GRID!' && python3 --version",
  "timeout_seconds": 30
}

Task results are stored in tasks/completed/{node_id}-{task_id}.json:

{
  "task_id": "task-001",
  "node_id": "worker-001",
  "exit_code": 0,
  "stdout": "Hello, D-GRID!\nPython 3.11.0",
  "stderr": "",
  "timestamp": "2025-10-16T10:01:23Z",
  "status": "success"
}

✨ Key Features

• Atomic Operations: Uses git mv for transactional task acquisition (no race conditions)
• Self-Healing: Auto-detects orphaned tasks and moves them back to queue
• Isolated Execution: Docker containers with strict security (network=none, read-only FS, limited processes)
• Real-Time Dashboard: Auto-generated, always up-to-date status view
• Zero Infrastructure: Works with free GitHub tier (no servers, no databases)
• Worker Autonomy: Nodes self-register and heartbeat without central coordination
• Full Transparency: All state visible in Git history

🔐 Security (Current Status)

MVP implementation with essential security:

• ✅ Container isolation (network=none, read-only, user=1000:1000)
• ✅ Fixed base image (python:3.11-alpine, not customizable)
• ✅ Output limits (10KB max per task to prevent DoS)
• ✅ JSON validation and required fields
• ✅ Task submission validation via GitHub Actions

Future hardening needed:

• Task signing & verification
• Enhanced container sandboxing
• Rate limiting per worker
• Multi-signature quorum for critical operations

� Quick Start

Prerequisites

• GitHub account and a public repository
• Docker installed (for running workers)
• Python 3.11+ (for local development)
• Git CLI

For Repository Maintainers

# 1. Create a new repository on GitHub
# 2. Clone it locally
git clone https://github.com/<your-username>/d-grid.git
cd d-grid

# 3. Enable GitHub Pages
#    Settings → Pages → Source: gh-pages branch

# 4. Enable GitHub Actions
#    Settings → Actions → Allow all actions

# 5. Create a GitHub Personal Access Token (if needed for private repos)
#    Settings → Developer settings → Personal access tokens
#    Keep this secret!

For Workers (Start a Node)

Using Docker (Recommended)

docker run -d \
  --name dgrid-worker-001 \
  -e NODE_ID=worker-001 \
  -e DGRID_REPO_URL=https://github.com/<your-username>/d-grid.git \
  -e GIT_TOKEN=ghp_your_token_here \
  -e PULL_INTERVAL=10 \
  -e LOG_LEVEL=INFO \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -p 8000:8000 \
  fabriziosalmi/d-grid-worker:latest

Then check the dashboard at http://localhost:8000

Using Docker Compose (Development)

# Clone the repository
git clone https://github.com/<your-username>/d-grid.git
cd d-grid

# Configure environment in docker-compose.yml
# Then start:
docker-compose up -d

# Check logs:
docker-compose logs -f dgrid-worker

Local Python (Development Only)

cd worker
pip install -r requirements.txt
python main.py

Submitting Tasks

1. Fork the repository

git clone https://github.com/<your-username>/d-grid.git
cd d-grid

2. Create a task file

cat > tasks/queue/my-first-task.json << 'EOF'
{
  "task_id": "my-first-task",
  "script": "echo 'Hello from D-GRID!' && python3 --version",
  "timeout_seconds": 30
}
EOF

3. Commit and push

git add tasks/queue/my-first-task.json
git commit -m "Add task: my-first-task"
git push origin main

4. Monitor execution

Visit your dashboard at: https://<your-username>.github.io/d-grid/

Results appear in tasks/completed/ or tasks/failed/ depending on outcome.

📊 Dashboard

Access your real-time dashboard at: https://<your-username>.github.io/d-grid/

Dashboard shows:

• 🟢 Active nodes (with last heartbeat time)
• ⏳ Pending tasks in queue
• 🔄 Currently executing tasks
• ✅ Successfully completed tasks
• ❌ Failed tasks

The dashboard updates every 60 seconds and regenerates every 5 minutes via GitHub Actions.

🛣️ Roadmap

Phase 1: ✅ Foundations (COMPLETE)

• Worker node core implementation
• State manager (node registration & heartbeat)
• Task runner with Docker execution
• Git coordination and atomic operations
• Robust error handling and recovery

Phase 2: 🚧 GitHub Actions & Automation (IN PROGRESS)

• Task submission validator
• Dashboard generator
• Orphan task cleanup
• Auto-scaling recommendations
• Performance metrics collection

Phase 3: 🎯 Deployment & Release

• Publish Docker image to registries
• Enhanced documentation
• Example tasks and templates
• Public launch

Phase 4: 🔮 Advanced Features (Future)

• End-to-end test suite
• Task dependency management
• Worker reputation system
• Task priority levels
• Distributed consensus for critical ops
• Web UI for task submission

🤝 Contributing

Everyone can contribute! The easiest way is to launch a worker node:

docker run -e DGRID_REPO_URL=https://github.com/<repo> \
           -e NODE_ID=my-worker \
           -v /var/run/docker.sock:/var/run/docker.sock \
           fabriziosalmi/d-grid-worker:latest

You can also:

• Submit tasks by creating JSON files in tasks/queue/
• Improve code by forking and submitting pull requests
• Report issues on the GitHub issue tracker
• Suggest improvements in discussions

Development Setup

# Clone and setup
git clone https://github.com/<your-fork>/d-grid.git
cd d-grid

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
cd worker
pip install -r requirements.txt

# Run locally (requires Docker)
python main.py

📚 Documentation

• Worker Configuration - All environment variables explained
• Task Format Specification - Create valid tasks
• Architecture Deep Dive - Understand how it works
• Development Progress - Track ongoing development

🐛 Troubleshooting

Worker won't start:

# Check configuration validation
LOG_LEVEL=DEBUG python worker/main.py

# Verify Docker is running
docker ps

# Check Git credentials (if private repo)
git config credential.helper

Tasks aren't being picked up:

# Check repository sync
cd /tmp/d-grid-repo && git log --oneline -5

# Verify task format
cat tasks/queue/*.json | python3 -m json.tool

# Check worker logs
docker logs dgrid-worker-001

Dashboard not updating:

• Verify GitHub Actions is enabled in Settings
• Check Actions tab for workflow run history
• Ensure GitHub Pages is configured in Settings

� Support & Community

• Issues: Report bugs on GitHub Issues
• Discussions: Ask questions in GitHub Discussions
• Wiki: Additional info in GitHub Wiki

�📝 License

MIT License - Feel free to use, experiment, and build your own D-GRID!

See LICENSE for full terms.

🎉 Acknowledgments

Built with:

• ❤️ Passion for decentralization
• 🔧 Python, Git, and Docker
• 🚀 GitHub's free tier (Actions, Pages, API)
• 🌟 Open source community

---

FAQ

Q: Does this work without Internet? A: No, it requires connectivity to GitHub. It's designed for cloud/internet deployments.

Q: Can I use this for production workloads? A: Currently MVP. For production, add the security hardening listed in the Security section.

Q: How many workers can I run? A: Unlimited! Each worker is independent and only needs the repo URL.

Q: What happens if GitHub is down? A: Workers will retry until connectivity is restored. No data is lost since Git stores everything.

Q: Can I run on Kubernetes? A: Yes! Deploy workers as Kubernetes jobs that pull tasks from the Git repo.

Q: How do I scale this? A: Simply run more worker containers. They auto-coordinate via Git atomic operations.

---

Mission Status: 🚀 BLITZ-LAUNCH OPERATION IN PROGRESS

Decentralization starts here.