Pantheon - Docker AI Runtime for RPG Companions

A self-contained Docker runtime for running quantized GGUF language models with FastAPI backend and an immersive RPG-themed web interface. Built for low-latency CPU inference on servers with SIMD-capable processors.

Features

✨ High-Performance CPU Inference

• llama.cpp with AVX2/F16C/native SIMD optimization (10-50x speedup)
• Multi-threaded inference with automatic CPU detection
• HTTP API with proper token limit enforcement

🎭 RPG Chatbot Interface

• Fantasy-themed dark UI with gold accents
• Persistent chat history (localStorage)
• Real-time conversation context awareness
• Server-side session management for long-term memory

🔧 Easy Deployment

• Docker Compose orchestration
• Three-container architecture: runtime, API, client
• Auto-detection of GGUF models in /models directory
• Bootstrap script for Debian 13 VM setup

Quick Start

Prerequisites

• Docker & Docker Compose
• 2GB+ free disk space
• GGUF quantized model file

1. Clone & Navigate

git clone https://github.com/yourusername/docker-ai-runtime.git
cd docker-ai-runtime/docker

2. Add Your Model

Place a .gguf file in pantheon-llama/models/:

cp /path/to/your-model.gguf pantheon-llama/models/

Tested models:

• Gemma-3-1B-Q8_0.gguf (2.4GB, ~5 sec response)
• Unsloth-Q4_K_M.gguf (smaller, faster)

3. Build & Run

cd pantheon-llama
./run_all.sh

The system will:

• Build llama.cpp with SIMD optimization
• Create API and client Docker images
• Start containers on ports 8080 (web UI) and 8081 (API)

4. Access

• Web UI: http://localhost:8080
• API: http://localhost:8081/api/generate

Architecture

┌─────────────────────────────────────────────────────┐
│                 Nginx (Client Container)             │
│                    Port 8080                         │
│            ⟳ Reverse proxy to API                    │
│            ⟳ Serves static web UI                    │
└─────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────┐
│              FastAPI (API Container)                 │
│                    Port 8081                         │
│   ⟳ Session management (server-side memory)        │
│   ⟳ System prompt injection                         │
│   ⟳ Proxies to llama-server                         │
└─────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────┐
│         llama-server (HTTP LLM Server)               │
│              Internal Port 8081                      │
│   ⟳ llama.cpp inference                             │
│   ⟳ Token limit enforcement                         │
│   ⟳ Multi-threaded CPU inference                    │
└─────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────┐
│            /models/your-model.gguf                   │
│         (Mounted from host filesystem)               │
└─────────────────────────────────────────────────────┘

API Endpoints

POST /api/generate

Generate text from a prompt.

Request:

{
  "prompt": "What is your name?",
  "max_tokens": 100,
  "session_id": "optional-uuid"
}

Response:

{
  "output": "My name is...",
  "tokens_predicted": 42,
  "session_id": "uuid"
}

POST /api/session

Create a new conversation session.

Response:

{
  "session_id": "uuid-string"
}

DELETE /api/session/{session_id}

Delete a conversation session.

Configuration

Environment Variables

Set in docker-compose.yml:

environment:
  LLAMA_THREADS: "14"      # CPU threads (auto-detected if unset)
  API_PORT: "8081"         # API port
  CLIENT_PORT: "8080"      # Web UI port

System Prompt

Edit the SYSTEM_PROMPT in server.py:

SYSTEM_PROMPT = """You are a mystical RPG companion. 
Remember user details and stay in character."""

Model Selection

Rename models to enable/disable:

# Disable a model
mv models/gemma3-1b-Q8_0.gguf models/gemma3-1b-Q8_0.gguf.disabled

# Enable it again
mv models/gemma3-1b-Q8_0.gguf.disabled models/gemma3-1b-Q8_0.gguf

Performance Tuning

Optimize Build

CMake flags in Dockerfile.runtime:

-DLLAMA_NATIVE=ON      # Auto-detect CPU features
-DLLAMA_AVX2=ON        # Explicit AVX2
-DLLAMA_F16C=ON        # FP16 conversion

Increase Threads

export LLAMA_THREADS=28  # or your CPU count
./run_all.sh

Larger Context Window

Increase MAX_MESSAGES in server.py to keep more conversation history.

Troubleshooting

Port Already in Use

# Change ports
API_PORT=9081 CLIENT_PORT=9080 ./run_all.sh

# Or kill existing containers
docker compose down

Model Not Found

# Check models directory
ls -lh docker/models/

# Ensure .gguf extension
ls docker/models/*.gguf

Slow Inference

1. Check SIMD flags in build:

   docker exec pantheon-api /opt/llama.cpp/build/bin/llama-cli --help | grep AVX

2. Verify threads are allocated:

   docker logs pantheon-api | grep "Using LLAMA_THREADS"

Memory Issues

• Use smaller quantization (Q4 instead of Q8)
• Reduce MAX_MESSAGES for shorter context

Development

Local Testing

# Test API directly
curl -X POST http://localhost:8081/api/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Hi","max_tokens":10}'

# Test with session
curl -X POST http://localhost:8081/api/session

View Logs

docker logs pantheon-api     # API logs
docker logs pantheon-client  # Nginx logs

Rebuild Images

cd docker/pantheon-llama
docker compose down
./run_all.sh

File Structure

docker-ai-runtime/
├── README.md
├── docker/
│   ├── models/                    # Mount point for GGUF files
│   │   └── your-model.gguf
│   └── pantheon-llama/
│       ├── Dockerfile.runtime     # llama.cpp build
│       ├── Dockerfile.api         # FastAPI container
│       ├── Dockerfile.client      # Nginx container
│       ├── server.py              # FastAPI app
│       ├── entrypoint.sh          # API startup (llama-server + FastAPI)
│       ├── nginx.conf             # Reverse proxy config
│       ├── docker-compose.yml     # Container orchestration
│       ├── run_all.sh             # Start services
│       ├── build_all.sh           # Build images
│       ├── bootstrap_deb13.sh     # VM setup
│       └── web/
│           ├── index.html         # Web UI
│           ├── client.js          # UI logic
│           └── style.css          # RPG theme

Known Limitations

• Single-threaded inference lock: One request at a time (can queue in HTTP layer)
• CPU-only: No GPU support (llama.cpp can be compiled with CUDA)
• RAM context: Conversation history limited by available memory
• Model size: Quantized 4B-8B models recommended for consumer hardware

Future Improvements

• Streaming responses (SSE)
• Multiple concurrent inference sessions
• GPU support (CUDA/ROCm)
• Model hot-swap without restart
• Persistent session storage (SQLite)
• Admin dashboard

License

MIT

References

• llama.cpp - C++ inference engine
• FastAPI - Web framework
• Nginx - Reverse proxy
• GGUF Format - Model format

Support

Issues? Check:

1. docker logs pantheon-api for API errors
2. Browser console (F12) for client-side errors
3. Ensure model file exists and is valid GGUF
4. Verify CPU supports AVX2: cat /proc/cpuinfo | grep avx2