perf(loader): UMA-gated eviction + parallel warmer + zero-copy uploads — 2-3.4x faster dense loads, ~6x faster A3B MoE loads on dGPUs; fix sequential sampler seed by ghazni101 · Pull Request #617 · warpfront/hipfire

ghazni101 · 2026-08-22T17:20:15Z

Summary

hipfire's model→VRAM load path was leaving most of its bandwidth on the
table for three stacked reasons, each found by measurement:

UMA-only page-cache eviction applied everywhere — every tensor
read ended with posix_fadvise(DONTNEED), forcing a full disk
re-read of the model on every load. Correct on unified-memory APUs
(Strix Halo etc.), pure loss on discrete GPUs.
Serial per-tensor pread — a single FADV_WILLNEED does not
saturate this box's btrfs md0 (~1.7 GB/s serial vs ~4.5 GB/s with
concurrent readers), so cold loads serialized behind disk latency.
Heap staging copy per tensor — pread into a fresh Vec, then
hipMemcpy from that copy: page-cache → heap → VRAM instead of
page-cache → VRAM.

Three commits, one per cause:

9842657f gate HFQ page-cache eviction on UMA: HfqFile
gains evict_page_cache (default true = historical/UMA behavior);
the qwen35 carrier disables it on non-UMA devices (both pp=1 and
pp>1 paths); HIPFIRE_PAGE_EVICTION=1 restores old behavior;
arch_is_uma() classifies gfx1103/1150/1151/1152. deepseek4's
OOM-sensitive pread+fadvise lifecycle is deliberately untouched.
9d51c825 parallel cache warmer + mincore gate: 4 worker threads
pread the data region into the page cache (16 MiB chunks) while the
loader uploads; a guard stops them on drop. Skipped entirely when
mincore shows ≥90 % of pages already resident.
c6edb843 zero-copy uploads: qwen35 keeps its mmap alive on
dGPUs and reads tensors as borrowed slices (tensor_data) so
hipMemcpy DMAs straight out of page-cache pages; pread fallback
preserves UMA behavior.

A fourth idea was tried and reverted: blocking MADV_POPULATE
before the upload loop measured identical totals (it relocates the same
soft-fault work into one serial stall). Documented in prepare()'s
comment so it isn't re-added blindly.

New tooling: scripts/bench_model_load.py measures hipfire and the
ROCmFPX llama.cpp fork with identical windows (hipfire: daemon spawn →
{"type":"loaded"} protocol response; rocmfpx: spawn → end-of-load log
marker, process killed at marker so prompt eval/gen are excluded).
Microbenchmarks: h2d_mmap_bench.rs, h2d_microbench.rs.

Round 2: A3B MoE loads + a latent eviction leak (follow-up campaign)

Deploying the branch to the Ornith-1.5-35B-A3B mq4r serving container
exposed three more issues, found by phase-profiling real daemon loads:

Zero-copy missed the MoE expert sweep (0493003f) — the
packed-MQ4 loader (try_load_packed_mq4_experts) still pread all
512 experts per blob into fresh heap Vecs before hipMemcpy. This
sweep was ~28 s of every ~31 s load on the A3B model. Fix:
HfqFile::data_range() + an in-file contiguity probe — when all
experts lie back-to-back and pages are resident, upload straight
out of one borrowed mmap slice; non-contiguous/overlay/UMA keep the
old paths.
Parallel reader evicted its own pages (761de5d9) —
read_job() called fadvise(DONTNEED) after every segment,
unconditionally, so any load routed through the lane-parallel
reader re-read the whole model from disk (~0.7 GB/s) on every
restart. This contradicted the carrier's dGPU keep-resident policy;
hfq.evicts_page_cache() is now threaded through
read_jobs_from_path/read_job.
Per-layer residency probing (d2072ddf) — the first cut of the
zero-copy gate called mostly_page_cached() (fresh map+mincore of
the full multi-GB file) once per layer ≈ 17 s per load of pure
checking. Memoized per process.

Also riding in this push (1cd7a890): sequential serve requests
seeded the sampler RNG from a compile-time constant (0x13579BDF),
making temp>0 output byte-identical across requests for the same
prompt on the noslots path (batch scheduler already keyed entropy per
attempt). generate() now takes a request_seed; the daemon derives
it via batch_rng_for_key(AttemptKey). Greedy temp-0 stays fully
deterministic — verified byte-identical pre/post change.

Measurements

gfx1100 (RX 7900 XTX), fresh daemon process per run, medians of ≥5 runs.
Shared-box caveat: another agent benchmarks the same GPU/disk, so
run-to-run noise is ±10–20 %; all deltas below exceed it.

Isolation microbenchmarks (h2d_mmap_bench, 15.7 GB warm file):
heap-src H2D 15–27 GB/s · mmap-src warm-PTEs 15.2 GB/s · mmap-src
cold-PTEs 4.0 GB/s · old pread path ~4–6 GB/s aggregate.

End-to-end load, hipfire daemon:

model	master	final	speedup
qwen3.5-9b (md5 `296092bf…`)	4338 ms	~1600–1900 ms	~2.4×
qwen3.5-27b (sha256 `ea615949…8576e`)	9901 ms*	~3900–4400 ms	~2.4×
qwen3.8-27b (md5 `2fb2edc2…`)	12 528 ms	~3650–3800 ms	~3.4×
ornith-1.5-35b-a3b.mq4r (18.8 GB, warm cache)**	~29 000–36 000 ms	~5 500 ms	~6×

*qwen3.5-27b baseline taken mid-campaign after the 9B fix landed; the
clean pre-fix number for the same lineage is the 3.8-27B A/B above
(master binary built from 80a572c8, interleaved runs, identical
weights).
**Round 2 measurement: container daemon start → /v1/models serving,
interleaved old/new binary under a GPU lock; weight-sweep phase alone
dropped ~28 s → ~2.7 s (host-read 0 ms = borrowed-mmap path engaged).

Reference point — ROCMFPX llama.cpp fork loading equivalent weights,
same measurement window:

model	rocmfpx steady-state median	hipfire final
Qwen3.5-9B Q4_K_M (md5 `d3842bd7…`)	2189 ms	2108 ms
Qwen3.5-27B Q4_K_M (md5 `a34c995b…`)	5908 ms	4308 ms

Traced phase timings post-fix (27B): embed 77 ms / 1.24 GB
(~16 GB/s), lm_head 26 ms, full 64-layer sweep ~1.1 s (~11 GB/s).
Remaining budget: fixed init (~0.5 s), post-KV tail (~0.6 s), disk state.

Verification

cargo test -p hipfire-runtime: 550 tests pass at each commit.
Repo MQ4 speed gates pass at commit time (prefill +86–91 % over
baseline floor, decode +9 %).
Coherence eyeballed on decoded output post-fix: 9B (Python
string-reverse function), 27B (mutex paragraph), 3.8-27B (17×23=391,
19×21=399 with correct reasoning chains).
Deployed validation: two production serving containers rebuilt from
this branch (ornith-mq4rp, qwen36-35b-mq4r, both retained-PM4
rigs) and verified through their real HTTP serving path — load,
greedy determinism, temp>0 sampling variation, and PM4 route
activation checked end-to-end after each deploy.
Sampler-seed fix verified by A/B: old binary reproduced byte-identical
completions 3/3 at temp 0.8; fixed binary produced distinct
generations 5/5, greedy unchanged.
All GPU measurements serialized through a gpu-coord flock protocol;
binary + model md5s recorded in the commit messages.

Notes for reviewers

Attempt ledger across the campaign: three failed attempts (eviction
gate initially landed only in the pp>1 path; the blocking-populate
experiment; the ungated first cut of the zero-copy gate, which let
cold-start H2D soft-fault serially off disk at ~0.25 GB/s — now
mincore-gated) — each documented with measurements rather than hidden.
Non-qwen35 archs keep default eviction unless their carrier opts in,
so no other load path changes behavior by default.
UMA systems are byte-for-byte on the old code path: mmap dropped,
fadvise per read, owned Vecs.
The sampler-seed commit touches behavior beyond loading; it is
included here because it was developed against and validated on this
branch's deployment targets. Happy to split it out if reviewers
prefer.

….75s on gfx1100 qwen3.5 single-GPU (pp=1) loads ran every tensor through pread + posix_fadvise(DONTNEED), evicting model pages from the page cache after each read and forcing a full disk re-read of the model on every load. That eviction exists for unified-memory APUs (Strix Halo etc.) where cached pages and hipMalloc'd VRAM staging share physical RAM — but it was applied unconditionally, including on discrete GPUs where the page cache is free bandwidth. - HfqFile gains evict_page_cache (default true = historical behavior; UMA paths unchanged) + set_evict_page_cache/evicts_page_cache. tensor_data_vec / tensor_data_pread / drop_pages_range skip fadvise when it is off; HIPFIRE_PAGE_EVICTION=1 forces the old behavior. - arch_is_uma() classifies gfx1103/1150/1151/1152 as UMA. - qwen35 carrier (both pp=1 and pp>1 paths) disables eviction on non-UMA devices before loading. - warm_page_cache(): one POSIX_FADV_WILLNEED over the data region so disk DMA overlaps GPU init and early uploads instead of serializing behind per-tensor reads. Measurements (Qwen3.5-9b.mq4 md5 296092bf1e6a45d78c1acf815eb93366, gfx1100 RX 7900 XTX, fresh daemon process per run, spawn→loaded JSON): baseline (master): median 4338 ms n=5 spread 3.6% attempt1 (eviction gate): median 4364 ms no change — first gate landed only in the pp>1 path, pp=1 was still evicting attempt1b (pp=1 gate): median 1805 ms -58% attempt2 (+WILLNEED warmup): median 1751 ms n=7 spread 3.4% Reference: ROCmFPX llama-completion loading unsloth Qwen3.5-9B-Q4_K_M.gguf (md5 d3842bd7fba0ff2e6771a89272c5b862), spawn→'main: llama threadpool init': median 1835 ms, n=5 spread 3.4%. Isolation evidence: microbench H2D pageable memcpy 27.5 GB/s, warm pread 6.4 GB/s — neither load-limiting once the page cache survives; the old path re-read from disk at ~1.3 GB/s effective. Verification: hipfire-runtime suite 550+8 tests pass; daemon smoke generate on the 9B produces coherent decoded text. New tooling: scripts/bench_model_load.py measures both engines with identical windows (hipfire: daemon spawn → {"type":"loaded"}; rocmfpx: spawn → 'llama threadpool init' log line, process killed at the marker so prompt eval/gen are excluded).

Follow-up to the eviction gate. Under real cache pressure the qwen35 load path was still disk-bound: serial per-tensor pread sustains only ~1.7 GB/s on this box's btrfs md0 (single FADV_WILLNEED does not saturate the array), while llama.cpp's mmap readahead overlaps reads with copies and wins. Measured: 6 concurrent readers reach ~4.5 GB/s. - start_cache_warmup(): 4 worker threads pread the data region into the page cache chunk-sequentially (16 MiB chunks) while the loader uploads tensors; CacheWarmerGuard stops them on drop. dGPU-only (no-op when page-cache eviction is on, i.e. UMA). - mostly_page_cached(): mincore over an untouched shared mapping; when ≥90% of pages are already resident the warmer skips (a fully cached file needs no re-read, and spawning workers costs ~0.3 s of CPU contention on 9B-class loads). Measurements (fresh daemon process per run, spawn→loaded): 27B (qwen3.5-27b.mq4 sha256 ea615949…8576e, 14.984 GB): pre-warmer median 9901 ms → post 4308 ms (n=5, spread 6.5%) ROCmFPX llama-completion, same-family weights (unsloth Qwen3.5-27B-Q4_K_M.gguf md5 a34c995b30e7ada6e89c8a3c0a0e479b, spawn→'llama threadpool init'): steady-state median 5908 ms. 9B (md5 296092bf…): 2108 ms vs rocmfpx steady-state 2189 ms. Verification: hipfire-runtime suite 550 tests pass; daemon smoke generate produces coherent decoded output on both 9B and 27B.

…s at 11-16 GB/s Follow-up to the eviction gate + parallel warmer. With page cache warm, the remaining wall was the heap staging copy: every tensor went page-cache -> pread Vec -> hipMemcpy. Microbench (h2d_mmap_bench, new): heap-src H2D 15-27 GB/s; mmap-src with warm PTEs 15.2 GB/s; mmap-src cold PTEs 4.0 GB/s (soft faults inside the copy loop); pread path ~4-6 GB/s aggregate. Changes: - qwen35 HfqSource::prepare keeps the mmap alive on dGPUs (gated on !evicts_page_cache, i.e. exactly where eviction is already disabled). - load_weight_tensor / read_embed / lm_head reads prefer the borrowed mmap slice (tensor_data) and fall back to pread on UMA (mmap dropped there) via a Cow helper. No owned per-tensor Vec on the hot path. - populate_mapping() was tried and REVERTED: a blocking MADV_POPULATE before the upload loop measured identical totals — it relocates the same soft-fault work into one serial stall. Documented in prepare()'s comment so nobody re-adds it blindly. Measured steady-state medians (fresh daemon process per run; shared box, other agent active -> +/-20% run-to-run noise): qwen3.5-9b: 1.6-1.9 s (was 2.11 with warmer only) qwen3.5-27b: 3.9-4.4 s (was 4.31) qwen3.8-27b: 3.65-3.80 s (was 4.31) Trace on 27B: embed 77 ms/1.24 GB (~16 GB/s), lm_head 26 ms, 64-layer sweep ~1.1 s (~11 GB/s). Remaining budget is fixed init (~0.5 s), post-KV tail (~0.6 s), and disk state. Verification: hipfire-runtime suite 550 tests pass.

Adding h2d_microbench/h2d_mmap_bench as auto-discovered examples pushed ungated_examples to 34 (ceiling 32). They are GPU-only tools by nature — exactly what required-features gating is for — so declare them under 'deltanet' like the other hardware-bound probes. ungated_examples back to 32; leanup-ratchets 21/21 OK locally.

check-crate-maps --check drifted on hipfire-arch-qwen35, hipfire-loader, hipfire-quantize and hipfire-runtime: the load-fix commits changed symbols/counts those generated blocks track. Regenerated; ratchets 21/21 OK, crate-maps 39/39 match.

… green The clippy (advisory) job exits 101 on current stable — pre-existing on master too (verified: pure 80a572c fails identically; toolchain bump to clippy 1.98 made these deny-level): - hipfire-ds4-parent codec.rs/gemm_ref.rs/layer_ref.rs test indices: '0 * bk + N' erasing_op patterns -> plain indices (row 0), with the block-coordinate kept in comments. - hipfire-ds4-parent gemm_ref.rs a_s[0 * 2 + 1] -> a_s[1]. - saddle-lab test_forward_slots_golden: DECODE_STEPS was left at 0 by a debugging session ('was 3' comment), making an empty range (reversed_empty_ranges is deny). Restored to 3, which re-enables the intended decode steps through forward_scratch — the canonical entry point per the note in the same file. cargo clippy --workspace --all-targets --locked now exits 0.

…d 0x13579BDF Sequential (noslots) serve requests on the retained-PM4 route replayed the identical xorshift stream for every request: ar::generate hardcoded rng_state = 0x13579BDF (qwen path) / 42 (llama path) and reset the process-global CPU sampler RNG to 0x13579BDF. With temp>0 every request with the same prompt produced byte-identical output, run to run. The sampler kernels themselves are launched raw-HIP and are never captured into the PM4 tape, so the frozen RNG came from seeding, not replay. Thread a request_seed parameter through generate() and derive it at the daemon call site from the same attempt key the continuous-batch scheduler already uses (batch_rng_for_key(AttemptKey)), which exists precisely to give 'each lane/request a distinct stream rather than the same fixed seed'. temp=0 greedy stays deterministic; batch/EP paths are untouched. Carries the rustfmt debt for the two touched files required by CI's changed-file fmt gate.

…cache The packed-expert sweep dominates A3B MoE load time (~28s of ~31s on ornith-1.5-35b-a3b.mq4r): every layer pread 512 expert tensors into fresh heap Vecs, then hipMemcpy'd them to VRAM — two full host copies per blob on top of the disk/page-cache read. c6edb84 wired zero-copy mmap uploads for dense tensors but missed this path. Add HfqFile::data_range() and a contiguity probe: when all experts of a blob lie back-to-back in-file and the pages are resident (mincore, >=90%), upload_raw copies straight out of the mapping — no staging Vec. Evicted-cache cold starts keep the parallel-pread path: a borrowed-slice H2D from disk soft-faults serially inside the copy loop (~0.25 GB/s), worse than the lane-parallel reader. Also log per-load sweep timing ('weight sweep: N ms') for ops visibility. Measured on gfx1100, warm page cache: weight sweep 28s -> 2.2s, daemon start-to-ready ~31s -> ~5s.

…ader read_job() called fadvise(DONTNEED) on every segment unconditionally, so any load routed through the lane-parallel reader — including the packed- MQ4 expert sweep that dominates A3B MoE loads — evicted its own pages and re-read the entire model from disk on every restart (~25s for an 18.8 GB mq4r on gfx1100), contradicting the carrier's dGPU keep-resident policy and keeping HfqFile::mostly_page_cached() false forever. Thread hfq.evicts_page_cache() through read_jobs_from_path/read_job so only UMA / explicit opt-in evicts.

mostly_page_cached() maps + mincores the whole model file (~0.4 s on an 18.8 GB mq4r) and was called once per layer — ~17 s of pure residency checking per load. Compute it once per process instead.

ghazni101 · 2026-08-23T03:43:51Z

Round 2 pushed to this branch (4 new commits, 1cd7a890..d2072ddf) — description updated accordingly.

New:

perf(qwen35): zero-copy packed-MQ4 expert uploads (HfqFile::data_range + contiguity probe + mincore-gated borrowed-slice H2D). Weight sweep on ornith-1.5-35b-a3b.mq4r: ~28 s → ~2.7 s; daemon start→ready ~31 s → ~5.5 s warm.
fix(runtime): the parallel HFQ reader called fadvise(DONTNEED) per segment unconditionally — it evicted its own pages and forced a full disk re-read every restart on dGPUs. Now honors evicts_page_cache.
fix(generate): sequential serve seeded the sampler from constant 0x13579BDF; same prompt was byte-identical across requests at temp>0. Now keyed per attempt like the batch scheduler; greedy unchanged (verified byte-identical).

All gates pass (fmt, MQ4 speed floor); both PM4 serving containers rebuilt and verified end-to-end.

hipfire-tune-agent added 11 commits August 22, 2026 15:12

chore: regenerate hipfire-ds4-parent crate map after clippy index fixes

3193062

perf(qwen35): memoize page-residency probe across the layer sweep

d2072dd

mostly_page_cached() maps + mincores the whole model file (~0.4 s on an 18.8 GB mq4r) and was called once per layer — ~17 s of pure residency checking per load. Compute it once per process instead.

ghazni101 marked this pull request as draft August 23, 2026 04:28

Navigation Menu

Uh oh!

perf(loader): UMA-gated eviction + parallel warmer + zero-copy uploads — 2-3.4x faster dense loads, ~6x faster A3B MoE loads on dGPUs; fix sequential sampler seed - #617

perf(loader): UMA-gated eviction + parallel warmer + zero-copy uploads — 2-3.4x faster dense loads, ~6x faster A3B MoE loads on dGPUs; fix sequential sampler seed#617
ghazni101 wants to merge 11 commits into
warpfront:masterfrom
ghazni101:perf/model-load-time

ghazni101 commented Aug 22, 2026 •

edited

Loading

Uh oh!

ghazni101 commented Aug 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

ghazni101 commented Aug 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Round 2: A3B MoE loads + a latent eviction leak (follow-up campaign)

Measurements

Verification

Notes for reviewers

Uh oh!

ghazni101 commented Aug 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ghazni101 commented Aug 22, 2026 •

edited

Loading