Step 25: Row-interleaved weight repacking — contiguous multi-row block layout #52

Roadmap Step

Step 25 — Row-interleaved weight repacking

Dependencies

• Depends on Step 14 (Q4_K_M dequantization — quantized weight format support)

Scope

At model load time, repack quantized weight matrices from row-major into row-interleaved format so SIMD lanes processing multiple output rows access contiguous memory instead of striding across rows.

• Row-interleaved format — For Q8_0_R8: take 8 consecutive rows' blocks and pack them contiguously. Block b of row r in the original layout at offset r * rowBytes + b * blockBytes maps to interleaved offset (b * 8 + r%8) * blockBytes + (r/8) * 8 * rowBytes. SIMD loads 8 blocks from 8 rows in one contiguous read.
• Format variants — Q8_0_R4 (4-row interleave, AVX2), Q8_0_R8 (8-row, AVX-512). Similarly for Q4_K, Q5_K, Q6_K, Q5_0 formats.
• Repacking at load time — After weight tensor metadata is resolved from GGUF, allocate a new NativeMemory.AlignedAlloc(64) buffer and repack. Original mmap'd data still available for fallback. Repacking cost amortized over inference.
• TransformerWeights integration — Add repacked weight pointer field alongside original weight pointer. Kernels check for repacked pointer first.
• M dimension alignment — Output dimension (M) must be padded to interleave factor (4 or 8). Handle tail rows with original non-interleaved kernel.
• Memory overhead — Repacked buffer is same size as original weights (just reordered). Total model memory ~2× during repacking, then original mmap can be released if not shared.

Key Files

• src/DotLLM.Cpu/Kernels/WeightRepacking.cs (new — repacking logic per quant format)
• src/DotLLM.Models/Architectures/TransformerWeights.cs (add repacked weight pointers, trigger repacking after load)
• src/DotLLM.Models/Architectures/TransformerModel.cs (use repacked weights in Gemm/Gemv dispatch)
• src/DotLLM.Cpu/Kernels/MatMul.cs (add repacked-format GEMV/GEMM entry points)

Tests

• tests/DotLLM.Tests.Unit/Cpu/Kernels/WeightRepackingTests.cs
• Test: repack then unpack roundtrip produces identical bytes
• Test: Q8_0_R4 GEMV produces identical output to standard Q8_0 GEMV
• Test: Q8_0_R8 GEMV produces identical output to standard Q8_0 GEMV
• Test: tail rows (M not divisible by interleave factor) handled correctly
• Test: repacked buffer alignment is 64-byte

References

• docs/QUANTIZATION.md — block layouts per quant type
• llamafile/tinyBLAS — row-interleaved weight format for SIMD matmul
• ik_llama.cpp — repacked weight formats for outer-product kernels

Acceptance Criteria

• WeightRepacking class implements repack/unpack for Q8_0, Q5_0, Q4_K, Q5_K, Q6_K
• Repacked weights allocated via NativeMemory.AlignedAlloc with 64-byte alignment
• TransformerWeights stores repacked pointers alongside originals
• Repacking triggered at model load time (one-time cost)
• Tail row handling for M not divisible by interleave factor
• GEMV/GEMM with repacked weights produces bit-exact output vs. original layout
• Unit tests for roundtrip correctness and numerical equivalence
• Memory cleanup: repacked buffers freed on model dispose