You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf(cpu): MoE decode (n=1) routed experts run on k single-thread Parallel.For tasks - two pool dispatches per layer instead #981
Found while investigating why CPU decode on MiMo-V2.6-Flash BPW2.0 (72 GiB mmap) is 1.1 tok/s vs llama.cpp 7.5.
Per-op breakdown (DOTLLM_CPU_PROF=1, warm page cache): ~90% of a decode token is the routed-expert FFN. MoeSwiGluMlp.ExecuteRoutedFromAssignments runs the k selected experts as k independent single-threaded tasks (Parallel.For, inner GEMMs without a pool), serial gate -> up -> down per expert, with ArrayPool rentals per task. Only k cores work, the ComputeThreadPool (spinning) is idle, and a .NET thread pool wake-up is paid per layer.
Fix: for seqLen==1 with a pool, flatten (expert x projection x row) and run two ComputeThreadPool dispatches per layer (gate+up, then down) through the same ComputeRows kernels the n=1 GEMM column uses -> bit-identical to the grouped path.
Acceptance: bit-identical unit test vs grouped path (IQ/K-quant/Q8_0), A/B tok/s per MoE model, env switch DOTLLM_CPU_MOE_DECODE_N1=0.
Activity
Merge pull request #982 from jamesburton/issue/981-moe-decode-n1
jamesburton commentedon Oct 13, 2026
Merged in #982