Skip to content

[qwen4exp T1] Expert tiering / cache (VRAM <-> RAM <-> SSD) (L, defer) #825

Open

Description

@jamesburton
  • Scope: only for GPU-addressable memory < model (UD-Q4_K_XL class on this box; dGPU). Design: provider abstraction (vLLM RFC), per-layer LRU/LFU-decay cache with async slot upload and id->slot remap (llama.cpp #27861), per-byte slot sizing (Strata finding 8), optional static warm-start profile (STRP format), --dump-routing tool.
  • Acceptance: gated by a measurement first: record routing traces (dotLLM CPU oracle) on 3 workloads and report per-layer LRU-64/128 hit rates and static-profile transfer; build only if > 60% hit at <= 15% of experts. On this box also measure heap-0 vs SSD-mmap fallback for a 111 GB file.
  • Deps: V1, M1 (verify windows touch more experts).

Context: Qwen3.8-Flash-Next (arch qwen4_exp, 176B/6B active, 48 layers = 36 GDN + 12 QSA, 512-expert top-10 softmax MoE + shared expert, 4-stream gated residual, n-gram table on layer 1, MTP head). Strata (Niko1221/Strata) is the performance target (measured on this box 2026-10-07: IQ2_XS 50 tok/s decode with MTP, ~450-620 prefill). Reference chain: HF transformers qwen4_exp, llama.cpp -dev none CPU (not Vulkan), KL via llama-perplexity. Epic: see parent.

Parent: #814

Activity

  1. jamesburton commented on Oct 13, 2026

    @jamesburton
    OwnerAuthor

    Stage 0/1 results (full numbers: .docs/EXPERT_TIERING_DESIGN_2026-10-13.md): gate PASSES, per-layer LRU-64 (12.5% of experts) = 70.7% decode hit (6 workloads), LRU-128 86.7%; but SSD random read ~1-1.7 GB/s => 3.4-3.7 tok/s ceiling at 64 slots, 7-9 at 128. Unbuffered reads into host-visible/imported memory = full device speed, no cache bloat; mmap+copy 2.5-4x slower. Static profiles do not transfer (27-48%). MTP verify window touches 31-33 experts/layer. Stage 2 design written; prototype not yet built.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions