Run giant LLMs on a normal PC — faster, smaller, without losing intelligence.
Alyxx Engine runs 860B+ parameter MoE models on consumer hardware (a 16 GB GPU), streaming the experts from disk through a native I/O pipeline built to squeeze every GB/s out of the NVMe, RAM and GPU — without degrading the model: the output is the same the full model would give (a fidelity gate is measured on every optimization).
🇪🇸 Léelo en español · License: BSL 1.1 (source-available)
Loading an 861.6B-parameter model normally needs a datacenter (~0.9 TB of VRAM even at its native FP8/FP4 precision). Alyxx runs it on 16 GB by treating the model like a streaming world (an ECS engine): only what each token needs is resident; the rest streams from an NVMe copy compressed to ~3 bits per weight (AQLM-3), decoded on the GPU on the fly. A native, GIL-free C reader and a pinned ring buffer keep the disk→RAM→VRAM pipe full.
The result is a giant model that is usable on hardware you already own — and it stays lossless.
Reference machine: RTX 4070 Ti SUPER (16 GB) · 32 GB RAM · NVMe Gen4. Model: DeepSeek-V4-Pro — an 861.6B MoE (61 layers × 384 experts), 581 GB on disk in AQLM-3 (the native FP8/FP4 checkpoint is 865 GB).
| Metric | Alyxx Engine | Naïve baseline |
|---|---|---|
| Decode — with the AQLM-3 pack | 2.46 s/tok record · typical band 2.5–3.0 on a rested machine | ~31 s/tok |
| Decode — stock checkpoint, no pack (native FP8/FP4 streaming) | 5.82 s/tok | ~31 s/tok |
| Prefill | ~1.0 s/tok | 3.34 s/tok |
| On-disk size | 581 GB (AQLM-3, ~33% smaller than the native FP8/FP4) | ~1.7 TB (BF16) |
| Fidelity | 100% on confident positions (202/202) · ppl +0.2% | — |
Small and mid models that fit run resident (CUDA-graph decode); dense models that don't fit stream too (a 7B in bf16 on a 16 GB card: 16.5× faster than the naïve path, still lossless).
- Any Hugging Face / GGUF model, stock. Dense or MoE, resident or streaming — works as
downloaded, nothing to convert.
alyxx.verifyreproduces the lossless proof on your own models, today. - Giant MoE from its original checkpoint. Streams at native FP8/FP4 precision — the 861.6B above runs like this at 5.82 s/tok with zero preparation. No AQLM-3 required.
- + AQLM-3 pack (optional turbo). ~3-bit experts alongside the checkpoint: 2.46 s/tok and 581 GB on disk. The calibrated converter ships in v0.2 — the converter currently in the repo validates the pipeline but does not yet meet the lossless bar (we say it before you measure it). Until v0.2, the pack is the one piece you can't easily build yourself.
The AQLM-3 overlay currently lives alongside the original checkpoint (both on disk); the self-contained converted format is part of the converter milestone.
# 0. Requirements: Python 3.11+, an NVIDIA GPU with CUDA 12.x drivers.
git clone https://github.com/Alejandro-GDL/Alyxx-Engine && cd Alyxx-Engine
# 1. Install torch with CUDA first (the generic PyPI wheel is CPU-only):
pip install torch --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt
pip install -e . # optional: gives you the `alyxx` command
# 2. Chat with any Hugging Face model directory (safetensors or GGUF).
# Everything runs stock — giant MoE included (native-precision streaming).
# An optional AQLM-3 pack alongside unlocks the fastest path (see Formats).
python -m alyxx.cli --model /path/to/model
# commands: /reset clears history · /raw <text> raw continuation
# /top <text> top-10 x-ray · salir quits
# flags: --greedy deterministic · --max-tokens N · --no-prefix-cache
# 3. Don't trust the "lossless" claim — verify it on your machine:
python -m alyxx.verify --model /path/to/model --quickFirst run compiles the GIL-free native reader once (zig cc, seconds, cached); if zig is
unavailable the engine falls back to a pure-Python I/O path — slower, still lossless. The
desktop app (chat + live-engine panel) is a developer preview: cd app && npm install && npm run dev (needs Node + Rust).
Honesty note: every number here was measured on Windows 11. The Linux/macOS code paths exist by design (each I/O primitive has a native API + fallback) but are untested on real machines — reports and PRs very welcome.
Alyxx is model-agnostic: it derives everything from the checkpoint topology (no per-model
ifs), so any transformer-decoder — dense or MoE, safetensors or GGUF — runs through one path.
Verified so far across seven architecture families — Llama, Mistral (verified
out-of-the-stable on release day: downloaded fresh, verify PASS, 6.4× from the arsenal, zero
tokens changed), Qwen2, Gemma2, GPT-NeoX, DeepSeek-V2/MLA and the DeepSeek-V4-class MoE — plus
qwen and gemma2 GGUF builds. Known limitation: checkpoints with fused projections (e.g.
Phi-3's packed qkv_proj/gate_up_proj) aren't unpacked by the topology discovery yet — caught
by our own pre-release drill; fix targeted for v0.1.1. If verify fails on your model, that's a
first-class bug: please open an issue with the model name.
"Lossless" here means as capable — verified, not promised. Don't trust us: run the proof on your machine, with your model:
python -m alyxx.verify --model /path/to/your/model # add --quick for a fast smokeIt generates with the full speed arsenal ON (twice — checking your stack is deterministic), then again with every speed lever kill-switched OFF (the slow classic reference), and compares the generated token ids bit by bit. Real output on the reference machine (gemma-2-2b):
regime ON: GPU OFF: GPU
speed ON: 10.04 tok/s OFF: 5.76 tok/s -> 1.7x from the arsenal, zero tokens changed
determinism (ON vs ON): OK — bit-identical
lossless (ON vs OFF): OK — bit-identical
VERDICT: PASS — the speed levers changed ZERO tokens on your machine.
The reference path is deliberately slow — that's the point. On models that don't fit in VRAM
the gap explodes: a 7B on the same machine measured 20.5× from the arsenal — zero tokens
changed. (Different baseline than the benchmark table: verify compares against the fully
kill-switched reference, a stricter bar than the naïve path.) The deeper lab gate also ships:
python tools/suite.py # the one-command test gate (needs the reference models)
python tools/probe_unbuf.py # bit-exact regression of the unbuffered native reader- Lossless — every optimization passes a fidelity gate (id canary + entropy-stratified silhouette) before it ships. Lossless = as capable, measured.
- Agnostic — to the model (all derived from checkpoint topology, zero
if model), to the hardware (budgets measured at runtime, never constants), and to the OS (Windows/Linux/mac, each primitive with its native API + fallback). - The engine is the product — no retraining, no distillation, no second model. Every lever works on the user's original weights.
alyxx/
├── runtime/ Engine: ECS orchestration, layer-major prefill, native v5-ring decode
├── model/ StreamingModel (attention MLA/GQA by capability, MoE, dense)
├── io/ AQLM-3 overlay (blob+index), GIL-free native C reader (NO_BUFFERING),
│ RAM-tier with a pre-touched slot-ring + MEM_RESET discipline
├── kernels/ Triton (fused dequant-SwiGLU for AQLM/FP4/R3) + C (native_read, built with zig)
├── moe/ Agnostic router + expert cache
├── codecs/ AQLM-3, rotated-3bit (R3), FP4
└── ecs/ The entity-component-system world of the runtime
Every speed lever is default-ON with a restore flag. The key ones: layer-major prefill (cost ~flat with prompt length), the native v5-ring reader (sector-aligned NO_BUFFERING to pre-touched pinned slots), and the memory discipline (post-load budget + MEM_RESET → zero page-out).
The face of the engine: a local 3-panel app (nav │ chat │ live engine) where you watch the
expert cascade, tok/s and residency in real time — the differentiator over a CLI or API. Plus a
model library and an AQLM-3 converter. Built with Tauri. In progress — see app/.
A prebuilt Windows installer (.msi) ships with each release. It is unsigned (young
source-available project): Windows SmartScreen will warn on first run — click More info →
Run anyway. Verify your download against the SHA-256 published in the release notes, or
build from source if you prefer.
- ✅ Engine — lossless, agnostic (safetensors + GGUF), record speed. Here today.
- 🔶 App — library, chat, live-engine panel with an honesty indicator; more UI landing.
- 🔶 AQLM-3 converter (calibrated) — turn any checkpoint into its ~3-bit pack at the lossless bar, from the UI. Target: v0.2.
- 🗺️ Second brain — an associative memory (kNN-LM) that makes the model smarter without training, gated to only speak up when confident. Engine-side done; UI next.
Business Source License 1.1 (source-available). You can read, run, modify and use it freely for personal, non-commercial purposes. Commercial use is governed by the license, and each version converts to Apache 2.0 two years after its release. Full terms in LICENSE and COMMERCIAL.md.
Built by Alejandro González De León. The research lab (probes, A/Bs, documented no-gos, the full story behind every law) lives separately; only what matures and passes the gates is promoted here.