Inference performance tuning

[m96-chan]

Goal

Improve token generation throughput (tok/s) and reduce GPU memory waste.

Known Issues

Rust (wgpu) backend: ~30 tok/s vs TS (WebGPU) ~60 tok/s

The Rust oxbitnet crate on the same hardware (RTX 5090) produces ~30 tok/s — half the throughput of the TypeScript/WebGPU path. Needs profiling to identify the bottleneck (dispatch overhead, buffer mapping, pipeline caching, etc.).

Uniform buffer leak

~900+ uniform buffers are created per forward pass and never released or pooled. Each createBuffer call is an IPC round-trip to the GPU process.

Fix: Pool or pre-allocate uniform buffers, reuse across passes.

GGUF file fetched twice

The model file is fetched once for weight loading and again for tokenizer extraction. On a ~700 MB file this doubles download time on first load.

Fix: Parse both weights and tokenizer metadata from a single fetch.

OpenGL ES fallback on Linux + NVIDIA + Wayland

Chrome WebGPU falls back to ANGLE OpenGL ES on this configuration (crbug/442791440), resulting in ~100x slower inference vs Vulkan.

Workaround: Launch Chrome with --ozone-platform=x11 to force X11 mode, which enables Vulkan WebGPU. Not a code fix — document in README/troubleshooting.

Optimization Opportunities

Shader-level

• vec4 / u32x4 loads in ternary matmul kernels (currently scalar)
• Shared memory tiling for better cache utilization
• Fused RMSNorm + quantize kernels to reduce dispatch count
• Warp-level reduction for softmax / RMSNorm

Runtime-level

• Fix uniform buffer leak (pool or pre-allocate)
• Single-fetch for GGUF (weights + tokenizer from one ArrayBuffer)
• Reduce bind group creation (extend existing cache to prefill path)
• Profile dispatch overhead — minimize encoder.finish() + queue.submit() calls per token

Rust (wgpu) specific

• Profile wgpu dispatch vs browser WebGPU — identify where the 2x gap comes from
• Check device.poll() frequency and blocking behavior
• Review buffer mapping strategy (staging buffers, map_async timing)
• Benchmark pipeline compilation / caching overhead

Measurement

• Add --benchmark flag to examples/node-cli (fixed prompt, measure prefill + decode tok/s)
• Add --benchmark flag to Rust chat example for comparison
• Log per-layer timing via timestamp queries (where supported)

Prior Work

Phase	What	Commit
1–2	Buffer pool, pre-allocated uniforms	d9a9819
3	Bind group caching (decode path)	1b169c1

References

• Phase 4 investigation notes: OpenGL ES fallback root cause on Linux/NVIDIA/Wayland
• Chrome bugs: crbug/442791440, crbug/475935650