fix(serve): multi-slot engine fits 24 GB cards — skip GDN S-tape, chunk prefill scratch, apply device visibility - #608
Open
alpineQ wants to merge 2 commits into
Open
Conversation
…i_slot_prefill_chunk
…nk prefill scratch, apply device visibility
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #602. Three fixes that together let
serve.multi_slotstart on a consumer 24 GB card:Rig::buildcalledPrefillBatchScratch::new(), which allocates the DDTree tree-verify S-tape sized[max_batch × n_v_heads × 128²](f32 + q8) — ~21 GB at 16k ctx. The slots path is plain prefill and never tree-verifies;new_opt's own docstring prescribescap_gdn_tape=falsefor exactly this case.n_slots × prefill_chunk, notcap_tokens. The scheduler was built around chunked prefill (its docstring, and the SP3 spec's "slot 1 is chunk-prefilling 256 tokens" example), butrun_loopsetchunk_size = cap_tokens, which both forced cap-sized scratch and let one long prompt monopolise a step.EngineConfig.prefill_chunk(fromserve.multi_slot_prefill_chunk, default 1024) now bounds both.apply_device_visibility, sohardware.devices/HIPFIRE_DEVICESwere ignored and the engine always took device 0 — the display card on a desktop.Result: qwen3.6-35b-a3b.mq4r at 2 slots × 16384 ctx runs in 23.4 GB on a 24 GB gfx1100 (any usable ctx OOM'd before), pinned to the configured device.
Stacked on #607 (which registers
serve.multi_slot_prefill_chunk) — review that first; this branch contains its commit.Which crate(s) does this touch?
crates/hipfire-arch-qwen35hipfire-cliserve,bench_concurrency,test_serve_concurrent—EngineConfigconstructors)Test plan
cargo build --release --workspace --features deltanetcleancargo test --lib --workspace --features deltanetpassestest_prefix_cache_equivalence,test_swap_roundtrip,test_swap_equivalence,test_serve_concurrent— all pass (none of them can even start on beta at this ctx: the engine OOMs first)attn_legacy_baseline.shbitwise-identical to beta (single-sequence path untouched)speed.arch-fastpasses in the gate belowchange_gate telemetry
unit.env-docs— 173 findings, every one in a crate this diff does not touch (quantize, gemma4, deepseek4, redline, ...). No finding names a file in this PR.unit.no-gpu-control—hipfire-tui'sbounded_run_stdout_fill_is_not_falsely_timed_outfails identically on pristine beta (timing-sensitive dashboard test).serve.agentic.a3b-fast—zero tokens emittedon pristine beta too.serve.battery.qwen35-4b/-9b— the route argv asks--thinking medwithmax_tokens180/300, whichserve_harness.pyrejects before touching the GPU (the harness defect PR fix(serve): emit Qwen reasoning content #572 documented).