Measurement-first research on local Mixture-of-Experts inference under a hardware contract. One free Google Colab T4. Every experiment pre-registered. Every negative result published.
Author. dzgreeno — principal investigator. Site. https://dzgreeno.github.io/colossus-lab/
This project was developed and organized with support from Notion.
Powered by Notion — Supported by Notion Labs, Inc.
Notion did not fund GPU time, did not review the measurements, and does not endorse the conclusions. Responsibility for every number and every inference is the author's alone.
This is not a model, a framework, or a wrapper. It is a laboratory: an instrument, a protocol, a set of pre-registered experiments, and the raw evidence they produced - including the evidence that killed our own ideas.
The scientific object is the inference runtime of a sparse (MoE) language model on hardware that almost everyone already owns: a single GPU with 16 GB or less. No training. No fine-tuning. No cluster. Runtime only.
The subject model is gpt-oss-20b (21B total / 3.6B active, 32 experts, top-4, 24 MoE layers),
served through llama.cpp as GGUF, because its MXFP4 kernels require sm_90 and our GPU is sm_75.
Every run in this repository aborts before it starts unless the host satisfies
HW-CONTRACT-v1.0-FREE-T4:
| field | value |
|---|---|
| GPU | NVIDIA Tesla T4, compute capability 7.5 |
| VRAM (board / usable) | 15,360 MiB / 14,912 MiB |
| floor enforced | 14,500 MiB |
| host RAM | ~12.7 GB, ~9.18 GB available |
| vCPU | 2 |
| tier | Colab free |
A measurement taken outside the contract is not a measurement. It is an anecdote.
Three laws, each fitted independently several times, each with the mechanism identified.
L1 - the offload penalty is additive and workload-independent.
T(n) = T0 + b*n milliseconds per token, where n is the number of MoE layers moved to the CPU.
Nine independent fits across three workloads and three KV precisions give
b in [13.74, 16.32] ms/token/layer, median 14.75, all R^2 >= 0.982.
The apparent workload sensitivity in the raw data is an artefact of the denominator, not physics.
L2 - the context footprint decomposes, and only part of it can be quantized.
dVRAM(+4096 tokens) = K/q + V, with K = 92 MiB reproduced at two different offload levels
(residual 1 MiB), V = 14 MiB with no CPU/GPU split and V = 44 MiB once a split exists -
a step, not a slope. Consequence: KV quantization delivers 68-88 % of its arithmetic promise,
not 100 %.
L3 - offloaded experts are served from the page cache, not from a host allocation.
9,594 MiB left VRAM while resident process memory moved 0.01 GB and page cache stayed flat at
~9.8 GB. The weights are mmap-ed. Therefore a warm page cache is a precondition of every
throughput number in this repository, and it is declared as such.
L2 has an architectural corollary that we did not look for. K = 92 MiB per 4096 tokens matches
the 96 MiB predicted by the model's 12 full-attention layers and is incompatible with the
192 MiB predicted if all 24 layers cached full context. The instrument recovers the alternating
sliding-window structure of the architecture from black-box VRAM readings alone.
Four runtime ideas were pre-registered, executed, and failed. They are documented as carefully as the successes, because a runtime idea that fails against the right comparator is a result.
| idea | claim | verdict |
|---|---|---|
| ATLAS | KV precision is a throughput knob | closed - within-cell spread 6.7 % vs repeat CV 2.5 %; f16 won 10/15 cells |
| PROMETHEUS | latency contracts cut reasoning tokens at equal quality | failed - +8.1 % tokens vs fixed_medium at identical accuracy |
| JANUS | cheap signals predict when to escalate | failed - AUC 0.236 / 0.500 / 0.167 / 0.583, i.e. chance |
| CHRONOS | conversation-memory policies save tokens | void twice - both runs failed their own validity gates |
The four failures share one cause, and it is expressible as an inequality. See
papers/W2_negative_results/.
docs/ the contract, the protocol, the audit, the graveyard, the forecast
docs/papers/ scientific papers website (arXiv-style static academic site)
specs/ one pre-registration per experiment, written before the run
notebooks/ NB0..NB5, the executable form of the specs (Colab free T4)
src/ colossus.py - the instrument (contract, run ids, server, metrics)
analysis/ audit.py and laws.py - every number in docs/ is re-derivable
results/ one immutable folder per RUN_ID, raw.jsonl never edited
papers/ one folder per manuscript, with claims traced to RUN_IDs
tools/ claimcheck.py - CI refuses claims without provenance
- Open
notebooks/NB0_GATE.ipynbin Colab, free T4 runtime. - It builds
llama.cppstatically forsm_75, downloads the GGUF, and runs five preflight gates. It aborts if any gate fails. Expect ~65 min for the first build; the archive is cached on Drive afterwards. - Run
NB1..NB5in order, one notebook per session. - Each notebook writes
results/<RUN_ID>/withraw.jsonl,manifest.json,summary.csv,logs/. python analysis/audit.pyre-derives every table indocs/06_*.
RUN_ID = <UTC-ISO-compact>_<EXP>_<tier>_<sha8>, for example
20260805T003642Z_EXP-HERMES-001_colab-free_965b5d1f.
INTEGRITY.md is binding, not decorative. Its core clause: no number appears in this repository
without a RUN_ID and an openable raw file, and measured / derived / predicted are never mixed in
the same sentence. CI enforces a mechanical subset of this.
This repository supersedes colossus-moe. See PROVENANCE.md for what was retracted and why.
Code: MIT (LICENSE). Documents, figures and data: CC BY 4.0 (LICENSE-DOCS).
Machine-readable metadata is in CITATION.cff. Human form:
dzgreeno. Colossus Lab: measurement-first research on local Mixture-of-Experts inference. Version 1.0.0, 2026. https://github.com/dzgreeno/colossus-lab
This project was developed and organized with support from Notion — supported by Notion Labs, Inc.
Powered by Notion — Supported by Notion Labs, Inc.
See ACKNOWLEDGMENTS.md for the exact scope of that support, and for what it does not mean.
See MIGRATION.md. Short version: the old repository is renamed, not deleted, its final
state is preserved in the tag v0.1.0-legacy-retracted, and every figure it published without a
RUN_ID is withdrawn in PROVENANCE.md.