Skip to content

fix(serve): multi-slot engine fits 24 GB cards — skip GDN S-tape, chunk prefill scratch, apply device visibility - #608

Open
alpineQ wants to merge 2 commits into
warpfront:betafrom
alpineQ:fix/multislot-24gb
Open

fix(serve): multi-slot engine fits 24 GB cards — skip GDN S-tape, chunk prefill scratch, apply device visibility#608
alpineQ wants to merge 2 commits into
warpfront:betafrom
alpineQ:fix/multislot-24gb

Conversation

@alpineQ

@alpineQ alpineQ commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #602. Three fixes that together let serve.multi_slot start on a consumer 24 GB card:

  1. Skip the GDN S-tape in the engine's prefill scratch. Rig::build called PrefillBatchScratch::new(), which allocates the DDTree tree-verify S-tape sized [max_batch × n_v_heads × 128²] (f32 + q8) — ~21 GB at 16k ctx. The slots path is plain prefill and never tree-verifies; new_opt's own docstring prescribes cap_gdn_tape=false for exactly this case.
  2. Size scratch by n_slots × prefill_chunk, not cap_tokens. The scheduler was built around chunked prefill (its docstring, and the SP3 spec's "slot 1 is chunk-prefilling 256 tokens" example), but run_loop set chunk_size = cap_tokens, which both forced cap-sized scratch and let one long prompt monopolise a step. EngineConfig.prefill_chunk (from serve.multi_slot_prefill_chunk, default 1024) now bounds both.
  3. Apply device visibility in serve. The engine runs in-process and never executed the daemon's apply_device_visibility, so hardware.devices / HIPFIRE_DEVICES were ignored and the engine always took device 0 — the display card on a desktop.

Result: qwen3.6-35b-a3b.mq4r at 2 slots × 16384 ctx runs in 23.4 GB on a 24 GB gfx1100 (any usable ctx OOM'd before), pinned to the configured device.

Stacked on #607 (which registers serve.multi_slot_prefill_chunk) — review that first; this branch contains its commit.

Which crate(s) does this touch?

  • crates/hipfire-arch-qwen35
  • examples / daemon (hipfire-cli serve, bench_concurrency, test_serve_concurrentEngineConfig constructors)

Test plan

  • cargo build --release --workspace --features deltanet clean
  • cargo test --lib --workspace --features deltanet passes
  • GPU gates on gfx1100 24 GB with qwen3.6-35b-a3b.mq4r: test_prefix_cache_equivalence, test_swap_roundtrip, test_swap_equivalence, test_serve_concurrent — all pass (none of them can even start on beta at this ctx: the engine OOMs first)
  • attn_legacy_baseline.sh bitwise-identical to beta (single-sequence path untouched)
  • Paired fresh-process decode bench beta-vs-branch (qwen3.5-4b.mq4, asym3 + graph, 1 warm + 2 measured per side): 175.7 / 175.6 vs 174.7 / 174.7 tok/s → −0.55%, inside the ±1–3% session band
  • speed.arch-fast passes in the gate below
  • Change gate run and telemetry pasted below
change_gate telemetry
**change_gate: FAIL**

host gfx=`gfx1100` rocm=`6.18.37` models_dir=`/home/alpineq/.hipfire/models` · `80a572c824dcbbcdddac1cc5f6d1a7e91d7c4dd6`..`0d50f8b1a7cb41fec550659dc53a76baa6f46ff3` dirty · est=14.05min actual=37.0s

### Routes RUN
| route                   | status | duration |
| ----------------------- | ------ | -------- |
| serve.battery.qwen35-4b | fail   | 0.1s     |
| serve.battery.qwen35-9b | fail   | 0.1s     |
| speed.arch-fast         | fail   | 19.5s    |
| unit.arch-qwen35        | pass   | 1.7s     |
| unit.diff-check         | pass   | 0.0s     |
| unit.env-docs           | fail   | 0.2s     |
| unit.hipfire-cli        | pass   | 1.9s     |
| unit.hipfire-config     | pass   | 0.1s     |
| unit.hipfire-runtime    | pass   | 10.8s    |
| unit.no-gpu-control     | fail   | 2.5s     |

### Routes NOT RUN
| route | reason |
| ----- | ------ |
| —     | —      |

_Blocked or excluded routes mean coverage is incomplete — this report is not an admission that unrun surfaces are safe._
Pre-existing failures on this host, verified against a clean `upstream/beta` worktree and NOT introduced here:
  • unit.env-docs — 173 findings, every one in a crate this diff does not touch (quantize, gemma4, deepseek4, redline, ...). No finding names a file in this PR.
  • unit.no-gpu-controlhipfire-tui's bounded_run_stdout_fill_is_not_falsely_timed_out fails identically on pristine beta (timing-sensitive dashboard test).
  • serve.agentic.a3b-fastzero tokens emitted on pristine beta too.
  • serve.battery.qwen35-4b / -9b — the route argv asks --thinking med with max_tokens 180/300, which serve_harness.py rejects before touching the GPU (the harness defect PR fix(serve): emit Qwen reasoning content #572 documented).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant