You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Scope: only for GPU-addressable memory < model (UD-Q4_K_XL class on this box; dGPU). Design: provider abstraction (vLLM RFC), per-layer LRU/LFU-decay cache with async slot upload and id->slot remap (llama.cpp #27861), per-byte slot sizing (Strata finding 8), optional static warm-start profile (STRP format), --dump-routing tool.
Acceptance: gated by a measurement first: record routing traces (dotLLM CPU oracle) on 3 workloads and report per-layer LRU-64/128 hit rates and static-profile transfer; build only if > 60% hit at <= 15% of experts. On this box also measure heap-0 vs SSD-mmap fallback for a 111 GB file.
Deps: V1, M1 (verify windows touch more experts).
Context: Qwen3.8-Flash-Next (arch qwen4_exp, 176B/6B active, 48 layers = 36 GDN + 12 QSA, 512-expert top-10 softmax MoE + shared expert, 4-stream gated residual, n-gram table on layer 1, MTP head). Strata (Niko1221/Strata) is the performance target (measured on this box 2026-10-07: IQ2_XS 50 tok/s decode with MTP, ~450-620 prefill). Reference chain: HF transformers qwen4_exp, llama.cpp -dev none CPU (not Vulkan), KL via llama-perplexity. Epic: see parent.
Activity
jamesburton commentedon Oct 13, 2026
Stage 0/1 results (full numbers: .docs/EXPERT_TIERING_DESIGN_2026-10-13.md): gate PASSES, per-layer LRU-64 (12.5% of experts) = 70.7% decode hit (6 workloads), LRU-128 86.7%; but SSD random read ~1-1.7 GB/s => 3.4-3.7 tok/s ceiling at 64 slots, 7-9 at 128. Unbuffered reads into host-visible/imported memory = full device speed, no cache bloat; mmap+copy 2.5-4x slower. Static profiles do not transfer (27-48%). MTP verify window touches 31-33 experts/layer. Stage 2 design written; prototype not yet built.