Step 20: Multi-threaded CPU inference (#36)

[kkokosa]

Closes #36

Summary

• Custom zero-alloc ComputeThreadPool — N-1 background workers + caller = N total threads, delegate*<nint, int, int, void> function pointer dispatch with stack-allocated context structs, ManualResetEventSlim + CountdownEvent synchronization. Zero managed allocations on the hot path.
• Row-partitioned GEMV/GEMM — All MatMul variants (Q8_0, F32, F16) accept optional ComputeThreadPool?. GEMV partitions M rows across threads (4-row aligned for VecDot batching). GEMM parallelizes the tile loop. Falls back to single-threaded when pool is null or below threshold (m < 32).
• Head-partitioned Attention — Pointer-based Execute overload distributes heads across threads, each with its own scores buffer from per-worker scratch memory. Falls back when pool is null or numHeads < 2.
• --threads CLI option — Default 1 (single-threaded), 0 = auto (Environment.ProcessorCount). Wired through ThreadingConfig → InferenceOptions → LlamaModel.
• Bit-identical determinism — Pure partitioning (no cross-thread reductions), each output element computed by exactly one thread. 26 new parallel correctness tests verify bit-identical output at any thread count.

Benchmarks

Machine: AMD Ryzen 7 5800HS (8C/16T), .NET 10.0.3, AVX2

GEMV (M=K=4096, single-token decode)

Kernel	1T	2T	4T	8T	Speedup@8T
GemvQ8_0	1,409 μs	731 μs	454 μs	310 μs	4.54×
GemvF32	2,605 μs	1,826 μs	1,681 μs	2,048 μs	1.27×

Q8_0 scales near-linearly — compute-bound, data fits L3 cache. F32 is memory-bandwidth bound at 64MB weight matrix.

GEMM (M=K=4096, N=64, prefill)

Kernel	1T	2T	4T	8T	Speedup@8T
GemmQ8_0	82.7 ms	53.8 ms	23.6 ms	16.4 ms	5.04×
GemmF32	114.1 ms	63.5 ms	29.0 ms	20.3 ms	5.62×

Tile parallelism gives each thread a cache-friendly working set — both formats scale well.

Attention (32 heads, 8 KV heads, headDim=64, seqKv=128)

Threads	Mean	Speedup
1	73.3 μs	1.00×
2	40.8 μs	1.80×
4	28.9 μs	2.53×
8	21.7 μs	3.37×

End-to-end model inference

SmolLM-135M Q8_0 (--threads 8, temperature 0, 50 tokens):

Performance Summary:
  Prompt eval:       3 tokens in    10.26 ms (  3.42 ms/tok, 292.4 tok/s)
  Generation:       50 tokens in   564.99 ms ( 11.30 ms/tok,  88.5 tok/s)
  Total:            53 tokens in   575.25 ms ( 10.85 ms/tok,  92.1 tok/s)

Llama-3.2-1B Q8_0 (--threads 8, temperature 0, 20 tokens):

Performance Summary:
  Prompt eval:       3 tokens in    55.24 ms ( 18.41 ms/tok,  54.3 tok/s)
  Generation:       20 tokens in  1061.43 ms ( 53.07 ms/tok,  18.8 tok/s)
  Total:            23 tokens in  1116.67 ms ( 48.55 ms/tok,  20.6 tok/s)

Files Changed

File	Change
src/DotLLM.Core/Configuration/ThreadingConfig.cs	New — readonly record struct for thread count config
src/DotLLM.Core/Configuration/InferenceOptions.cs	Add Threading property
src/DotLLM.Cpu/Threading/ComputeThreadPool.cs	New — Zero-alloc thread pool with delegate* dispatch
src/DotLLM.Cpu/Kernels/MatMul.cs	+444 lines — parallel overloads for all GEMV/GEMM variants
src/DotLLM.Cpu/Kernels/Attention.cs	+122 lines — head-parallel pointer-based Execute
src/DotLLM.Models/Architectures/LlamaModel.cs	Store pool, wire to kernels, threading-aware LoadFromGguf
src/DotLLM.Cli/Commands/RunCommand.cs	--threads CLI option
tests/.../MatMulParallelTests.cs	New — 18 parallel correctness tests
tests/.../AttentionParallelTests.cs	New — 8 parallel correctness tests
benchmarks/.../ParallelBenchmarks.cs	New — Thread-scaling benchmarks
docs/ROADMAP.md	Step 20 ✅
README.md	Phase 2 progress 2/11, News entry

Test plan

• All 289 tests pass (dotnet test) — 263 existing + 26 new parallel correctness tests
• Bit-identical output: single-threaded vs multi-threaded for all GEMV, GEMM, Attention variants
• Below-threshold fallback: small matrices (m < 32) produce correct results via single-threaded path
• Null pool fallback: passing null produces identical results to explicit single-threaded
• End-to-end: --threads 1 vs --threads 8 produce identical token output (temperature 0)
• Zero managed allocations confirmed via MemoryDiagnoser in all benchmarks

🤖 Generated with Claude Code

[kkokosa]

@codex please review this PR thoroughly - performance issues and possible improvements, potential bugs, code quality and architecture

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[kkokosa]

@gemini please review this PR thoroughly - performance issues and possible improvements, potential bugs, code quality and architecture

[github-actions]

Gemini ✦

This is an exceptionally well-engineered PR. The architectural choices—specifically the zero-allocation ComputeThreadPool, delegate* dispatch, and stack-allocated context structs—are exactly what a high-performance .NET inference engine needs. The bit-identical determinism is also a massive win for testability.

I have reviewed the code for correctness, performance, and .NET best practices. Below are a few critical safety catches and minor performance suggestions.

🚨 Critical: Worker Exception Deadlock

In ComputeThreadPool.cs, if a worker thread throws an exception (e.g., an IndexOutOfRangeException or AccessViolationException during unmanaged memory access), the thread will terminate without signaling _completion. This will cause the caller thread (Thread 0) to deadlock forever on _completion.Wait().

Fix: Wrap the worker execution in a try/finally block. Ideally, catch and store the exception to rethrow it on the main thread, but at a minimum, ensure the barrier is signaled.

// src/DotLLM.Cpu/Threading/ComputeThreadPool.cs - Line 126
try
{
    _workFn(_context, threadIdx, _threadCount);
}
catch (Exception ex)
{
    // Optional: Store 'ex' in a shared array to rethrow on the caller thread
    Environment.FailFast("Compute worker crashed", ex); 
}
finally
{
    _completion.Signal();
}

⚡ Performance & Architecture Notes

1. Thread Priority:
   To prevent the OS scheduler from preempting your worker threads (which causes severe tail-latency spikes during barrier synchronization like _completion.Wait()), consider bumping the thread priority during creation:

   _workers[i] = new Thread(WorkerLoop)
   {
       IsBackground = true,
       Name = $"dotLLM-compute-{workerIdx}",
       Priority = ThreadPriority.AboveNormal // <-- Add this
   };

2. Excellent Cache-Line & SIMD Alignment:
   Your PartitionRows logic in MatMul.cs is brilliant:

   int chunk = ((totalRows + threadCount - 1) / threadCount + 3) & ~3;

   Forcing the chunk size to be a multiple of 4 ensures that threads don't step on each other's toes during 4-row unrolled SIMD loops, and it minimizes false sharing at the boundaries.

3. GQA-Friendly Attention Partitioning:
   In Attention.cs, partitioning by headsPerThread naturally keeps KV heads grouped together for Grouped-Query Attention (GQA). Because kvH = h / ctx.GroupSize, threads will share the same KV cache reads in L1/L2 cache. Very elegant.

4. ArrayPool LOH Allocation (GemmQ8_0):

   byte[] rented = ArrayPool<byte>.Shared.Rent(totalQ8Bytes);

   For a prefill of $N=64$ and $K=4096$, totalQ8Bytes is ~262 KB. This exceeds the 85 KB threshold and will be allocated on the Large Object Heap (LOH). ArrayPool handles LOH arrays fine, and since you return it, there's zero GC pressure. Just confirming this is safe and correct as written.

5. Scratch Buffer Thread Safety:
   In Attention.cs and MatMul.cs, Thread 0 calls pool.GetWorkerScratch(i, ...) for all threads before calling pool.Dispatch(). Because the workers are blocked on _workReady.Wait() while this happens, there are no data races on the _workerScratch arrays. This is a clever way to handle dynamic resizing without requiring locks inside the hot path.

🧹 Minor Code Quality Tweaks

• ComputeThreadPool.Dispose(): You are correctly checking _workerScratch[i] != 0 before freeing. Since nint defaults to 0, this is perfectly safe.
• Spin Count: new ManualResetEventSlim(false, spinCount: 100) is a good choice. The .NET default is typically 10-35 depending on the core count. Bumping it to 100 keeps the thread spinning slightly longer before yielding to the kernel, which is ideal for microsecond-scale inference loops.

Verdict: Outstanding work. Add the try/finally in the worker loop to prevent deadlocks on failure, and this is good to merge!