Step 20: Multi-threaded CPU inference — parallel GEMV, attention, and FFN #36

Roadmap Step

Step 20 — Multi-threaded CPU inference

Dependencies

• Depends on Step 6 (Llama forward pass — complete single-threaded pipeline)

Scope

• Parallel GEMV/GEMM — partition output rows across threads in MatMul. Each thread computes a subset of output elements independently (no shared mutable state)
• Parallel attention — distribute attention head computation across threads. Each thread processes a subset of heads (heads are fully independent in GQA)
• Parallel FFN — parallelize gate/up projections and down projection across output dimensions
• Thread count configuration — InferenceOptions.ThreadCount (default: Environment.ProcessorCount). Thread pool or dedicated threads for compute-bound work
• Work partitioning strategy — static partitioning with minimum chunk size to avoid excessive scheduling overhead. Use Parallel.For with MaxDegreeOfParallelism or a custom partitioner for cache-friendly chunks
• Thread-local scratch buffers — each thread gets its own scratch buffer from ArrayPool to avoid contention
• Target: ~4-8× speedup on multi-core CPUs for single-sequence decode

Key Files

• src/DotLLM.Cpu/Kernels/MatMul.cs (parallelize output rows)
• src/DotLLM.Cpu/Kernels/Attention.cs (parallelize across heads)
• src/DotLLM.Cpu/Kernels/RmsNorm.cs (potentially parallelize for large hidden dims)
• src/DotLLM.Core/Configuration/InferenceOptions.cs (add ThreadCount)

Tests

• tests/DotLLM.Tests.Unit/Cpu/Kernels/ParallelMatMulTests.cs
• tests/DotLLM.Tests.Unit/Cpu/Kernels/ParallelAttentionTests.cs
• Test: multi-threaded output matches single-threaded output exactly (deterministic)
• Test: thread count = 1 produces identical results to current implementation
• Benchmark: scaling test across 1, 2, 4, 8 threads

References

• llama.cpp ggml/src/ggml-cpu/ggml-cpu.c — thread pool and work partitioning
• docs/ARCHITECTURE.md — data flow and hot path identification

Acceptance Criteria

• GEMV/GEMM parallelized across output rows with configurable thread count
• Attention computation parallelized across heads
• InferenceOptions.ThreadCount configurable (default: Environment.ProcessorCount)
• Multi-threaded output is deterministic and matches single-threaded exactly
• Thread-local scratch buffers via ArrayPool (no contention)
• Minimum chunk size to avoid scheduling overhead on small matrices
• No lock contention on hot path (partition, don't synchronize)
• Benchmark shows scaling improvement (target: 4-8× on 8+ core CPU)
• ThreadCount=1 produces identical results (no regression)
• No thread safety issues in shared read-only data (weights, config)