Engineering Intelligence Hub

A production-style RAG service that answers engineering questions over internal technical docs (RFCs, runbooks, ADRs, API references) and cites the source passage for every claim. Runs free and offline by default (Ollama + open-source embeddings); swap one env var for Claude. Ships with a friendly web UI.

Also a learning companion: a built-in pipeline inspector shows exactly how each answer is produced — text → tokens → chunks → retrieval scores → prompt → grounded answer — so the RAG isn't a black box. See docs/HOW-RAG-WORKS.md.

All bundled documents are synthetic and fictional. Answers reflect the indexed docs and may be stale — verify against the live system before acting.

Capabilities

Grounded answers + citations	Every claim maps to a retrieved passage (F03)
Hybrid retrieval + rerank	BM25 + dense fused with RRF (F16), optional cross-encoder (F17)
OCR ingestion	Scanned PDFs + images via local Tesseract (F20)
Cleaning pipeline	Normalize, strip headers/footers, dedupe chunks (F21)
Hardened prompts	Prompt-injection resistant system prompt (F21)
Open corpus fetch	Real arXiv cs.SE / cs.DC abstracts, redistributable (F22)
Pipeline inspector	explain=true returns a full trace; UI renders it (F23)
Local-first	Ollama (GPU) default, no keys; cloud (Claude+Voyage) optional

What it does

Point it at a folder of technical documents (RFCs, runbooks, ADRs, API references, wiki pages). Ask a question in plain language. It retrieves the most relevant passages and asks an LLM to answer using only those passages, returning the answer plus the sources it relied on — so every claim is traceable. It refuses to answer when the documents don't cover the question, and preserves exact commands/endpoints verbatim.

Architecture

                ┌──────────────┐
  documents ──► │   ingest     │ load → chunk → embed → Chroma (vector DB)
                └──────────────┘
                       │
  question ──► retrieve top-k ──► prompt + LLM ──► answer + citations[]
                       ▲                                   │
                       └──────────── web UI (Next.js) ◄────┘

• Backend: FastAPI (/v1/ask, /v1/ask/stream, /v1/ingest, /v1/sources, /health, /ready, /metrics)
• Models (PROVIDER switch): ollama (llama3.1 + bge-small-en-v1.5, default, free) or claude (claude-opus-4-8 + Voyage voyage-3.5)
• Frontend: Next.js app in web/ — Ask page + "What's New" page
• Synthetic data: generated by scripts/generate_synthetic_data.py
• Quality: eval suite (hit-rate + faithfulness), pytest (offline), Docker, CI

See docs/ for the full kit: architecture, feature specs, security, runbook, definition of done.

Quick start (backend)

cp .env.example .env                          # defaults to the free Ollama provider
pip install -r requirements.txt
ollama pull llama3.1:8b                        # one-time, for the default provider
python scripts/generate_synthetic_data.py      # (re)generate the sample corpus
python -m app.ingest                           # build the vector index from data/
uvicorn app.main:app --reload                  # http://localhost:8000  (/docs for OpenAPI)

curl -s localhost:8000/v1/ask -H 'content-type: application/json' \
  -d '{"question":"How do I roll back the most recent deploy?"}'

Runs fully local by default — LOCAL_ONLY=true refuses the cloud path even if PROVIDER=claude. To deliberately use Claude, set LOCAL_ONLY=false, PROVIDER=claude, ANTHROPIC_API_KEY, VOYAGE_API_KEY.

OCR + real corpus (optional):

bash scripts/install-ocr.sh          # Tesseract + Poppler (Linux; Windows notes in docs/INGESTION.md)
python scripts/fetch_corpus.py       # pull real, open arXiv cs.SE / cs.DC abstracts into data/corpus/
python -m app.ingest                 # OCRs scanned PDFs/images and re-indexes

See the pipeline: ask with explain=true, or click 🔍 inspect pipeline in the UI:

curl -s localhost:8000/v1/ask -H 'content-type: application/json' \
  -d '{"question":"How do I roll back the most recent deploy?","explain":true}' | jq .trace

Running on Windows + GPU: this repo is developed on Linux but runs great on a Windows machine with an NVIDIA RTX GPU. Ollama uses the GPU automatically; install Tesseract (UB-Mannheim) + Poppler and add them to PATH for OCR. Full walkthrough: docs/INGESTION.md.

Run it on a new device

Two independent paths — Docker or native (no Docker) — each in CPU or GPU mode, on Windows/RTX, Linux, or macOS. Full matrix + walkthroughs in docs/DEPLOYMENT.md.

# Native, no Docker — one command bootstraps a venv + deps
./scripts/setup.sh            # Linux/macOS (CPU)      | --gpu for CUDA torch
.\scripts\setup.ps1 -Gpu      # Windows/RTX (CUDA)     | omit -Gpu for CPU

# Docker
docker compose up --build                                             # CPU (default)
docker compose -f docker-compose.yml -f docker-compose.gpu.yml up --build   # GPU (NVIDIA)

GPU is opt-in: set EMBED_DEVICE / RERANK_DEVICE to auto (cuda→mps→cpu), or force cuda / cpu / mps. The GPU compose overlay sets them to cuda automatically.

Quick start (frontend)

cd web && cp .env.example .env.local && npm install && npm run dev   # :3000

Develop

make setup      # install dev deps
make test       # offline tests (fake provider, no network)
make lint       # ruff
make typecheck  # mypy
make eval       # retrieval hit-rate + faithfulness (needs a provider)
make docker     # api + ollama via docker compose (CPU)
make docker-gpu # api + ollama with NVIDIA GPU (compose GPU overlay)

Disclaimer

Answers reflect the indexed docs and may be out of date — verify against the live system before acting on anything destructive. The bundled data/ is synthetic.

License

MIT — see LICENSE.