An engineering showcase. This repository documents the architecture, design decisions, and engineering process behind Allan — a private voice AI assistant for Windows. The full implementation is not public; what follows is a faithful, redacted account of how the system was built and why.
Author: Nick Siegel · Status: active private project · License: MIT (this showcase)
Lineage. Allan did not start 2.5 months ago. It is the embodied branch of a longer research arc — SYNTHAI, a self-teaching local cognitive-agent scaffold I built from my own theories of controlled machine self-improvement. SYNTHAI is the mind (memory, planning, reflection, benchmark-gated learning, and a from-scratch model lab); Allan is that mind given a voice, senses, and hands on a real computer. The design doctrine you'll see below — earn every change with evidence; propose offline, promote only proven winners; never let the system rewrite its own beliefs unchecked — originates in SYNTHAI. See Timeline → Lineage.
Allan is a voice assistant that runs entirely on a local machine — no cloud API in the default path. You hold a key and talk; it transcribes locally, reasons with a local LLM, and then acts: it controls native Windows applications through the accessibility tree, drives a browser, reads the screen with a vision model, and answers from a local knowledge base. It replies in a cloned voice. The hard part was never "call an LLM" — it was making a multi-process, GPU-contended, hardware- dependent system reliable enough to trust, and making a 7-billion-parameter model pick the right tool every time on an 8GB consumer GPU it has to share with a live text-to-speech engine.
The current Allan codebase alone is ~81,500 lines of Python across 433 modules, 202 test files (~1,700 tests), and 251 commits. That figure is just the embodied application — it sits on top of the older SYNTHAI research lineage it descends from.
Cloud voice assistants have three structural problems for a power user:
- Privacy — everything you say and everything on your screen leaves your machine.
- Capability ceiling — they answer questions; they don't operate your computer. They can't open your app, click the button, read your local notes, or run a multi-step task on your desktop.
- Latency and availability — they fail when the network does, and every turn is a round trip.
Allan is the opposite bet: local-first, action-oriented, private by construction. The cost of that bet is that you inherit every hard problem the cloud was hiding — model quality on small hardware, GPU memory contention, process supervision, crash recovery — and you have to solve them yourself. That's what this repository is about.
Most "AI assistant" projects are a thin wrapper around a hosted model. Allan is a systems project that happens to have an LLM at its core. The interesting engineering lives in the seams:
- A real agent loop, modeled as a finite state machine. The turn is not a
for-loop with four hiddenifbranches — it's an explicitTurnStateenum with a purestep()transition function, so "we ran tools but got no spoken summary" is a named state you can test, not an implied one. See ADR 0001. - A single dispatch seam for every capability. Every action the model can take is a
@tool-decorated function behind one registry. Nothing calls a tool directly — the registry is the only place logging, the security allowlist, and the kill switch are enforced. - Reliability engineering under a genuine hardware constraint. The cloned-voice TTS and the LLM must coexist on one consumer GPU. That single fact drives a subprocess supervisor with a circuit breaker (ADR 0003), a two-mode resource governor that frees VRAM when idle, an out-of-process daemon that restarts a dead app, and an evidence-gated model-promotion system so a model can only take over a route if it proves it fits (ADR 0005).
- Fastest-first perception. "Click Save" tries the accessibility tree first (<100ms), falls back to the browser DOM, then synthetic input, and only then to a vision model reading pixels. The cheap, reliable path is always tried before the expensive, fuzzy one.
- Honesty as an invariant. If the voice engine is down, Allan says so in text — it never silently substitutes a different voice. If a tool fails, the model sees the failure and reacts, rather than the error being swallowed.
flowchart TD
U([You hold-to-talk and speak]) --> STT[faster-whisper<br/>STT · local · CUDA]
STT --> TURN{{Agent turn · FSM}}
MEM[(Local knowledge · RAG<br/>+ working memory)] -. context .-> TURN
TURN -->|pick a tool?| REG[Tool registry<br/>single guarded seam]
REG --> ACT[Act on Windows / browser / screen]
ACT -->|result| TURN
TURN -->|final reply| TTS[SoVITS cloned voice<br/>TTS · supervised process]
TTS --> SPK([Speakers])
classDef local fill:#0f172a,stroke:#38bdf8,color:#e2e8f0;
class STT,TURN,REG,ACT,TTS,MEM local;
Every stage is local. The only network traffic is whatever a browsing task explicitly requests.
Full detail — with six more diagrams (turn sequence, perception chain, threading model, memory tiers, supervisor state machine): Architecture documentation →
Allan runs as an in-process desktop app — a native Edge WebView surface plus an always-available companion orb, not a web dashboard. These are real captures of the running app (default "Orrery" black-glass theme). The panels aren't decoration: they surface the actual runtime — live VRAM budget, per-route model assignments, subsystem health, and the memory graph.
The full workspace. Voice is the center; every other panel is a window you show, hide, resize, or
snap. Visible here: the Neural Map (the memory/concept "second brain"), the Capability Map
(routing topology with Allan at the center), Watch (a live telemetry log — bridge connected to Allan backend, stt ready, tts ready), and Settings:
Focus layout — the systems on display. The Watch panel shows the live VRAM budget (2.7 / 8 GB)
and subsystem status dots (ollama · sovits · worker · wake); Orchestrate shows the real
route→model table (chat → gemma3:4b, deep/fast_tool → qwen2.5:7b, ctx 4096, warm/cold state):
The Brain view — Allan's memory as a graph. Concepts, memories, wiki pages, recipes, sessions, and tools as linked nodes. Selecting a node (here, routing) shows its connections — the same gated-memory graph the architecture describes:
Focus, uncluttered. With panels hidden, the workspace is a calm orrery — the companion orb docked at the bottom, everything else a keystroke away:
Themed and configurable. A dark-panel arrangement with the settings drawer open (workspace mode, background, companion opacity, shortcuts) — and a fully separate "Retro 95" skin, because the UI is a real theming system, not a fixed screen:
More captures — the Retro 95 skin's typography/accent controls and additional layouts — are in
screenshots/.
| Concern | Choice | Why |
|---|---|---|
| Language / env | Python 3.11, uv |
Fast, reproducible single-venv installs; the AI/audio ecosystem is Python-native |
| Reasoning ("brain") | Ollama — qwen2.5:7b (tools/reasoning) + gemma3:4b (chat) | Local, swappable, fits an 8GB-class GPU; promoted only via a benchmark gate |
| Speech-to-text | faster-whisper (small.en, CUDA) | Accurate, fast, fully local |
| Text-to-speech | SoVITS cloned voice (supervised) | A distinct, personal voice; runs as its own supervised process |
| Vision | qwen2.5vl:3b | Reads the screen when the accessibility tree can't |
| Native control | uiautomation |
The accessibility tree is faster and more reliable than pixels |
| Browser | Playwright | Deterministic DOM control |
| Knowledge | sqlite-vec + nomic-embed-text | Local vector search; no external DB |
External runtimes (Ollama, faster-whisper, SoVITS, Playwright) are integrated, not authored here. Everything in the sections below — the agent, the tool system, the reliability layer, the memory system, the routing and evidence gate — is original work.
| Document | What it covers |
|---|---|
| Engineering Story | The narrative: problem → constraints → decisions → failures → iterations → what works |
| Architecture | Components, data flow, state management, failure recovery, scalability, tradeoffs, diagrams |
| Architecture Decision Records | Five load-bearing decisions, each with options, tradeoffs, and reasoning |
| Evidence | Real benchmark numbers, the test gate, screenshots — what actually works, honestly |
| Timeline | The SYNTHAI → Allan lineage and how the system evolved across 251 commits |
| Code snippets | Representative, sanitized examples of the core patterns |
| Demo | Runnable mini-demo — drives the core loop end-to-end with a fake brain |
| Interview Prep | The hard questions a reviewer would ask, answered honestly |
| Portfolio Summary | Ready-to-use blurbs for GitHub, resume, LinkedIn, applications |
| Repository Audit | A self-scored hiring-lens review with prioritized improvements |
This showcase follows one rule strictly: nothing here is fabricated. Every benchmark number is copied from a dated run in the private repository; every architectural claim maps to code that exists. Where a detail is inferred rather than measured, it says so. Proprietary specifics (exact prompt text, personal voice-source data, some internal thresholds) are omitted or shown as representative simplifications — the engineering is intact, the secrets are not.