Skip to content

perf(cpu): MoE decode (n=1) routed experts run on k single-thread Parallel.For tasks - two pool dispatches per layer instead #981

Closed

Description

@jamesburton

Found while investigating why CPU decode on MiMo-V2.6-Flash BPW2.0 (72 GiB mmap) is 1.1 tok/s vs llama.cpp 7.5.

Per-op breakdown (DOTLLM_CPU_PROF=1, warm page cache): ~90% of a decode token is the routed-expert FFN. MoeSwiGluMlp.ExecuteRoutedFromAssignments runs the k selected experts as k independent single-threaded tasks (Parallel.For, inner GEMMs without a pool), serial gate -> up -> down per expert, with ArrayPool rentals per task. Only k cores work, the ComputeThreadPool (spinning) is idle, and a .NET thread pool wake-up is paid per layer.

Fix: for seqLen==1 with a pool, flatten (expert x projection x row) and run two ComputeThreadPool dispatches per layer (gate+up, then down) through the same ComputeRows kernels the n=1 GEMM column uses -> bit-identical to the grouped path.

Acceptance: bit-identical unit test vs grouped path (IQ/K-quant/Q8_0), A/B tok/s per MoE model, env switch DOTLLM_CPU_MOE_DECODE_N1=0.

Activity

  1. added a commit that references this issue on Oct 13, 2026
  2. jamesburton commented on Oct 13, 2026

    @jamesburton
    OwnerAuthor

    Merged in #982

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions