refactor: unify device mesh, model parallelism, and generation dispatch by fivetide · Pull Request #527 · warpfront/hipfire

fivetide · 2026-07-12T18:56:58Z

Summary

This draft is the cumulative device-mesh and generation-driver refactor. It introduces a named-axis DeviceMesh, manifest-driven weight/state placement, dense TP and PP serving, MoE EP/TP execution work, and collapses the daemon's parallelism and architecture-specific generation ownership.

The branch is intentionally opened as a draft because important real-hardware validation remains unavailable locally. In particular, we need contributor help with physical multi-GPU PP and RCCL-backed EP validation before this should be considered production-ready.

Current status

Area	Status	Evidence
Device mesh and parallel ownership	Implemented	`DeviceMesh`, manifest-derived placement, and `ModelParallel` own topology and execution mode
Generation lifecycle	COR-002 and COR-003 complete	Total reset, terminal parser finalization, discard handling, and architecture lifecycle coverage
DeltaNet Step/manifest adoption	STEP-001 complete	Qwen35 manifest resolution and raw-vs-Step parity
MoE Step/manifest adoption	STEP-002 complete	`b1d54d5c2`; Gate 3: `GATE3_APPROVED_WITH_DEFERRED_DEBT`
Capability admission	CAP-001 complete	`10d29940`; source-aware admission and dense-EP normalization
Optional TP × EP composition	COMP-001 complete	Explicitly refused at every boundary
Final merge gate	Pending	Remaining AXIS, GEN, SPEC, VL, hardware, and DOC-002 tasks are tracked below

This PR remains a cumulative draft. STEP-002 is closed, but the PR does not claim arbitrary mesh composition or production Qwen35 TP/EP. Physical RCCL and remaining multi-GPU validation are still required before final merge.

Delivered changes

Added hipfire-hardware as the topology and collective owner.
Added named-axis DeviceMesh topology and stage/device grouping.
Added architecture weight/state manifests and manifest-derived placement/collectives.
Added dense tensor-parallel loading, forward execution, serving, LCP, and teardown.
Added dense pipeline-parallel loading, per-stage weight/KV residency, serving, and teardown.
Added DeepSeek4 and MiniMax expert-parallel execution paths.
Added reproducible fixed-point MoE down accumulation for partition-invariant TP/EP results.
Collapsed daemon AR serving onto ArchDispatch + ar_generate for most non-vision text paths.
Added StreamParser as the shared EOS/output/reasoning/tool-event hook.
Collapsed parallel state into ModelParallel::{Single,Tp,Pp,Ep}.
Collapsed LoadedModel from roughly 50 loose fields to 9 grouped fields.
Unified architecture-specific DeepSeek/MiniMax EOS storage into ModelMeta::eos_tok.
Completed COR-004 Qwen35 eviction lifecycle safety: model-owned CASK policy and reusable GPU scratch, request-owned compaction/recurrent/mirror state, and transactional DFlash/sidecar rollback coverage.
Completed STEP-002 MoE Step/manifest adoption across DeepSeek4, MiniMax, and Qwen35, including sealed lowering, fail-closed authority checks, and deterministic TP/EP parity harnesses.

Architecture support

Legend: Yes means implemented through the unified path; Partial means new ownership but bespoke execution remains; Target means technically appropriate but not implemented; N/A means the axis does not apply.

Architecture	Traits	Unified text AR	PP	TP	EP	Grammar/tooling	EOS/output hooks	Main gap
LLaMA/Mistral, arch `0`	Dense, standard attention	Yes	Yes, `PpModel` + `DenseDispatch`	Partial: only `has_qk_norm=true` / Qwen3-family metadata-eligible artifacts; non-qk-norm LLaMA/Mistral is refused	N/A	No token grammar; generic textual extraction only	Shared terminator/default parser	Broaden TP eligibility; `HW-006` physical closure for eligible artifacts
Plain Qwen3 through LLaMA carrier, arch `1`	Dense, standard attention, qk-norm	Yes	Yes	Yes	N/A	Same as LLaMA path	Shared EOS/terminator/default parser	Physical TP/PP validation breadth
Qwen3.5/3.6 dense, arch `5`	Dense, DeltaNet + full attention; optional VL	Yes, text only	Partial: `Pp(ArchResident)`, still `generate_multi`	Target	N/A	Qwen grammar matcher and tool calls	Shared parser, EOS, stops, think/checkpoint/KV hooks	Fold arch-resident PP; VL remains bespoke
Qwen3.5/3.6 MoE, arch `6`	MoE, DeltaNet + attention	Yes, text only	Partial	Target	Target	Same unified Qwen text grammar/tooling	Same shared text hooks	Choose TP/EP strategy; finish MoE/DeltaNet Steps
Dedicated Qwen2/VibeThinker, arch `7`	Dense, standard attention	Yes	Unsupported today	Target	N/A	No grammar/tooling; wrapper still drops user stops	Architecture-specific EOS set	Fix prompt framing, stops, and sampling
dots.ocr, arch `8`	Vision + Qwen2 decoder	No	Unsupported	Unsupported	N/A	No common grammar/tool parser	Bespoke EOS and greedy loop	Unify post-image-prefill lifecycle
DeepSeek V4, arch `9`	MoE, MLA/DSML, MTP/DSpark	EP: Yes; single-GPU fallback remains separate	Unsupported	Unsupported as TP	Yes	EP grammar matcher + DSML parser	DSML-specific parser; pre-commit EOS stop	Fold single-GPU fallback; align spec behavior
MiniMax M2, arch `10`	MoE, non-recurrent	Yes	Unsupported	Unsupported as TP	Yes	No grammar; prompt-side tool support	Shared parser with `[e~[` filtering	RCCL/physical EP validation
LFM2.5-MoE, arch `11`	MoE, recurrent convolution + GQA	Yes	Unsupported	Unsupported	Target	No grammar/tooling	Architecture stop-token family	EP and recurrent-state manifest work
Cohere2-MoE, arch `12`	MoE, agentic marker protocol	Yes	Unsupported	Unsupported	Target	Custom shared parser emits structured tool calls	Marker/forced-token/empty-turn hooks	EP implementation and MoE Steps
Qwen3.5-VL extension	Vision-conditioned Qwen35	No	Current `pp > 1` HFQ path bypasses VL detection and does not explicitly refuse at load; no PP+VL support claim	Unsupported	N/A	No grammar/tools in VL loop	Bespoke EOS/sampling/session handling	CAP-001 explicit admission error; share post-prefill lifecycle
Dense TP/PP runtime path	LLaMA/Qwen3 dense	Yes, `DenseDispatch`	Yes	Yes	N/A	No grammar	Shared AR lifecycle and LCP	No multi-GPU speculation

Cross-cutting generation status

Concern	Current status	Remaining issue
AR loop	Unified for most non-vision text paths	Qwen35 arch-resident PP, DeepSeek single-GPU fallback, and vision remain separate
Grammar	Generic `GrammarMatcher` hook	Real matchers currently exist mainly for Qwen35 and DeepSeek
Tooling	Generic parser/event surface	Qwen2, MiniMax, LFM, and VL do not have equivalent semantics
EOS	Shared `is_eos`, `EosDecision`, and filtering hooks	Literal markers require architecture-specific filters and tests
Stop sequences	Implemented in the common parser	Some wrappers still pass an empty stop list
Output parsing	Unified `StreamParser`	Vision and native Qwen MTP bypass it
Sampling	Common configuration	Backends still vary between GPU, CPU full-distribution, and forced argmax
Session commits	Centralized in `ar_generate`	Vision and native MTP retain separate lifecycles
LCP	Supported by the common lifecycle	Rewind policy remains architecture-specific; some models cold-reset
Reset	Broad canonical reset exists	Still contains a concrete model-state ladder rather than compiler-total dispatch
Speculation	Generic `generate_spec` core	Separate from AR; native Qwen MTP remains another loop
Multi-GPU speculation	Not supported	EP/TP speculative drafts are explicitly refused
Step execution	Core infrastructure exists	MoE, DeltaNet, recurrent, and vision adoption remains incomplete

Parallelism implementation

Pipeline parallelism

There are two implementations:

Dense PP is unified through PpModel, DenseDispatch, and ar_generate.
Qwen35 arch-resident PP is owned by ModelParallel::Pp, but generation still uses generate_multi.

Qwen35 cannot simply pretend to be dense PP because DeltaNet state and recurrent scratch are stage-resident. It should still be possible to share the request lifecycle through a Qwen35 mesh dispatch.

Tensor parallelism

TP is structurally unified but currently focused on dense LLaMA/Qwen3 execution. Qwen35 requires DeltaNet and attention projection placement to become first-class Step/manifest operations. MoE families need explicit TP-of-experts or EP semantics rather than reuse of dense TP by accident.

Expert parallelism

EP is implemented for DeepSeek4 and MiniMax. It is applicable but absent for Qwen35-MoE, LFM2-MoE, and Cohere2-MoE.

The earlier EP manifest replication defect is fixed: non-expert weights and state now use the complete compute grid for their owning PP stage.

Speculation implementation

Speculation is more unified than some historical design notes suggest. generate_spec is generic over:

Speculator
carrier-provided SpecTargetGuard
carrier-provided SpecEmit

This covers substantial portions of DFlash, DDTree, DSpark/MTP, and model-free n-gram speculation.

It is nevertheless a separate engine from ar_generate:

ArchDispatch::as_spec_target() is effectively dead; target acquisition uses Carrier::spec_target_guard.
Native Qwen MTP remains a standalone generation loop.
Vision has no normal speculative route.
Dense TP/PP and EP do not support speculative drafts.
Sampling capability differs by speculator.
DeepSeek single-GPU speculation does not yet share all AR stop/think behavior.

The likely clean end state is one request-lifecycle orchestrator with separate AR and speculative strategies, not one giant merged token loop.

Verification

STEP-002 CPU and static checks

MiniMax: 25 tests passed.
DeepSeek4: 114 tests passed.
Qwen35 default suite: 408 passed, 19 ignored.
Qwen35 harness suite: 470 passed, 21 ignored.
Runtime MoE-plan suite: 136 passed.
Post-cleanup focused suites: MoE plan 136/136 and weight manifest 76/76 passed.
Changed-file rustfmt and git diff --check passed.
Package-wide clippy remains nonzero because of the accepted untouched baseline; all 28 STEP-002 diagnostics in moe_plan.rs and weight_manifest.rs were resolved.

STEP-002 hardware and coherence

DeepSeek TP1/TP2 batched and MTP parity passed; decode and prefill delta: 0.00e0.
DeepSeek MTP coherence passed with 80 coherent committed tokens.
DFlash coherence passed 4/4 with no soft flags.
Named peer-direct EP2 gates passed:
- DeepSeek FNV: 0x26a13602bedf9926
- MiniMax FNV: 0x887c2e7717e9c3bf
Qwen35's EP2 probe is test-only parity evidence. Production Qwen35 TP/EP remains planned and refused under AXIS-002.

Earlier branch gates still relevant

Full workspace library tests passed after the generation-driver consolidation.
COR-004 DFlash coherence and default/CASK-sidecar multi-turn gates passed.
CAP-001 deterministic Qwen3.5-9B one-replica parity passed on gfx1151/HIP 7.2 for 25 committed token IDs.

Remaining work and known limitations

Hardware validation blockers

DeepSeek4 and MiniMax EP need complete RCCL-backed generation runs after the final ModelParallel::Ep migration.
Dense PP needs validation on two physical GPUs, including actual per-stage allocation, peer copies, parity, capacity gain, unload, and reload.
Qwen35 arch-resident PP needs the same physical residency and transfer validation.
TP teardown/reload should be repeated on final integration head on physical hardware.

Structural gaps

AR and speculation are still separate top-level engines.
Qwen35 arch-resident PP remains outside ar_generate.
DeepSeek4 single-GPU fallback remains separate.
Qwen35-VL and dots.ocr remain bespoke.
Universal Step/manifest forward execution is incomplete.
TP×EP is explicitly refused (COMP-001).
Reset ownership is not yet compiler-total through ArchDispatch.
Qwen35 CASK policy and reusable GPU scratch are model-owned; compaction cursor and recurrent/DFlash mirrors reset per request. Generic target loading (COR-005) is complete; eviction physical-cap allocation (COR-006) is complete.

Explicitly deferred STEP-002 debt

DeepSeek scalar inter versus inter_local cleanup.
Pre-existing gfx1151 depth-1500 compressed-KV recall artifact.

Contributor help wanted: AMD multi-GPU validation

We specifically need contributors with supported AMD multi-GPU systems and RCCL installed.

Please comment with your available topology before starting so runs are not duplicated. Useful systems include:

2 or more RDNA3/RDNA4 GPUs for dense PP/TP and Qwen35 PP.
2 or more CDNA GPUs for EP and collective validation.
A host with working /opt/rocm/lib/librccl.so.1 for DeepSeek4/MiniMax EP.
Heterogeneous AMD GPU pairs for stage-boundary and peer-copy testing.

Requested validation:

DeepSeek4 EP-2 and MiniMax EP-2

Load through the daemon's ModelParallel::Ep path.
Generate coherent output to completion.
Exercise DeepSeek grammar/tool calls and MiniMax stop sequences.
Run at least two turns and verify no state bleed.
Unload/reload repeatedly and report VRAM drift.
Include GPU models, ROCm/RCCL versions, model artifact hashes, full command, and logs.

Dense PP-2 and Qwen35 PP-2

Verify weights and KV allocations are physically split by stage.
Verify peer access and stage-boundary transfers.
Compare output against a single-GPU or known-good baseline where the model fits.
Report whether PP enables a model/context that does not fit on one GPU.
Run unload/reload cycles and report VRAM drift.
Run scripts/store-pp-gate.sh where applicable.

Dense TP-2

Run decode, prefill, multi-turn, and unload/reload parity examples.
Confirm collective behavior on physical devices rather than emulated aliases.
Report numerical deltas and token parity separately.

Please attach raw logs. Performance numbers without model hashes, prompt hashes, GPU topology, and ROCm/RCCL versions are not actionable.

Authoritative work tracker

The authoritative tracker is feature/device-mesh/.agent-progress/device-mesh-refactor-tracker.md. If this PR body disagrees with the tracker, the tracker wins and this body must be corrected.

PAR-003 is retired and must not be reused. COMP-001 owns the optional TP × EP composition decision; DOC-002 is the final validation and merge gate.

Completed

Dependency-driven implementation

Axis implementation

AXIS-001 Implement Standard-Attention PP/TP Cells
AXIS-002 Implement Qwen35 TP And MoE EP Cells
AXIS-003 Implement Additional Non-Qwen35 PP/TP/EP Cells
AXIS-004 Implement Vision-Family PP/TP Cells

Physical validation and final gate

Review guide

This is a large cumulative refactor and should be reviewed by subsystem: topology/hardware, manifests/loading, TP, PP, EP/MoE, generation dispatch, and model-state ownership.
Historical progress notes are included for forensic context and contain intermediate states. The local tracker is the sole authority for current status; this PR body and checklist are a non-authoritative contributor mirror and must be synchronized to the tracker.
This PR does not claim arbitrary mesh composition. Current supported configurations are single-axis modes plus PP stage grids; TP×EP is explicitly refused (COMP-001) with a defense-in-depth rejection before the EP-wins remap.

taniguchi-taku-softm · 2026-07-14T15:15:56Z

Physical PP Validation Findings on 2× R9700 — Partial, Blocked

Date: 2026-07-14
PR: #527
Commit: 4df035373669369484797abdd274f3f710c4c061

Environment

Item	Value
GPU	2× AMD Radeon AI PRO R9700 (gfx1201, RDNA4)
Interconnect	PCIe Gen5 x8 (2 hops, weight 40), same NUMA node
P2P	✅ Enabled (bidirectional, verified via `rocm-smi --showtopoaccess` and `gpus.enable_peer_all()` in `pp_parity`)
VRAM	32 GiB each (64 GiB total)
ROCm	7.2.4
amdgpu kernel driver	6.16.13
Kernel	6.17.0-35-generic
RCCL	2.27.7.70204
HIPFIRE branch	`feature/device-mesh` (PR#527)

Artifact fingerprints

File	Size	SHA-256
`qwen3.5-9b.mq4` (arch 5, 32 layers, DeltaNet)	5.3 GB	`ba83acf5bfd5d4e334b0afc26d779734e31623bb7f74e807c3581dfecb3128ad`
`qwen3-0.6b.hf4` (arch 1, 28 layers, plain Qwen3 via LLaMA carrier)	436 MB	`7760b19dfb940f8b33078eb524602b4f2b5e6825c6e10c466e6e99bcfc133838`

Note: HW-003's requirement specifies qwen3-0.6b-llama.mq4 (arch 0, pure LLaMA). That exact artifact is not on disk — tested with qwen3-0.6b.hf4 (arch 1, plain Qwen3 via LLaMA carrier) instead.

Results

1. VRAM split: `pp2_vram_probe` ✅ PASS (Qwen35, physical PP-2)

qwen3.5-9b.mq4: layers=32, dim=4096, hidden=12288, vocab=248320, kv_heads=4, head_dim=256

── per-card VRAM deltas (PP=2 split) ──
        baseline:  dev0= 0.000 GiB  dev1= 0.000 GiB
       Δ weights:  dev0= 2.638 GiB  dev1= 2.638 GiB
       Δ scratch:  dev0= 0.013 GiB  dev1= 0.013 GiB
            Δ KV:  dev0= 0.134 GiB  dev1= 0.134 GiB
      Δ DN state:  dev0= 0.006 GiB  dev1= 0.006 GiB
───────────────────────────────────────
           total:  ~2.8 GiB / card    (8.7% of 32 GiB)

Equal per-stage weight, KV, and DeltaNet state allocation confirmed

2. Physical peer access ✅ Confirmed

rocm-smi --showtopoaccess reports bidirectional P2P. gpus.enable_peer_all() in pp_parity succeeds without error. Actual stage-boundary data transfer (via boundary_copy in the PP forward path) executed implicitly during pp_parity run — no explicit transfer-trace log was captured.

3. Token parity: `pp_parity` ❌ (58/100 match, Qwen35 PP=1 vs PP=2)

PP=1 (single GPU) vs PP=2 (real 2 GPU) greedy decode (temp=0):

First 20 tokens: identical
[5328, 1301, 279, 2614, 1414, 321, 4087, 279, 3296, 3559, 13, 271, 1138, 25, 198, 760, 2614, 369, 264, 11782]
58/100 tokens match → first divergence at index 58
- PP=1: [... 3694, 314, 29687, 13, 198, 760, 3788]
- PP=2: [... 3694, 314, 6824, 11, 30954, 11, 321]

Working hypothesis: numerical differences introduced around the DeltaNet stage-boundary state transfer, which may be amplified by recurrent state evolution. This has not yet been isolated from possible synchronization, copy, or state-ownership defects.

Note on HW-004 closure: HW-004 requires PP=2 output to byte-match single-device cold generation and a two-turn fixture by hash. This result does not satisfy either criterion — the divergence is unresolved, not "expected."

4. Load/unload cycles: `pp_unload_reload` ✅ PASS (100 cycles, emulated PP)

LLaMA model (qwen3-0.6b.hf4, arch 1, emulated PP-2 via HIPFIRE_EMULATE_GPUS=2), 100 load→unload cycles with per-cycle VRAM monitoring:

VRAM baseline: 0.101 GiB
cycle    0: loaded=0.487 GiB  after_unload=0.231 GiB  drift=0.130 GiB
cycle   49: loaded=0.487 GiB  after_unload=0.231 GiB  drift=0.130 GiB
cycle   99: loaded=0.487 GiB  after_unload=0.231 GiB  drift=0.130 GiB
───────────────────────────────────────────────────────────
VRAM final:   0.231 GiB
total drift:  0.130 GiB (zero growth after cycle 0)

130 MiB drift occurs on first cycle only, then stays flat for 100 cycles — one-time HIP runtime context + kernel code object overhead, not a leak
Zero panics or crashes across 100 cycles

Caveat: This test exercises the load_model_pp + unload_model code path under emulation, not physical per-device teardown. HW-003 requires physical per-GPU VRAM recovery to within 64 MiB of post-first-unload baseline with no monotonic increase across cycles 2–4. That has not been tested.

5. Manifest placement parity: `store-pp-gate.sh` ❌ Forward failed (emulated PP)

llama_store_pp example under HIPFIRE_EMULATE_GPUS=2:

llama_store_pp: PP-2 placement OK — 155 tensors on stage 0, 156 on stage 1
  (embed→0, output-norm+lm_head→1, 28 layers banded by stage_for_layer)

Placement: ✅ Correct manifest-driven stage split confirmed (bit-exact tensor placement)
Forward: ❌ hipMemcpy D2H: an illegal memory access was encountered at forward_logits → scratch.logits download failed
Root cause: LLaMA forward path (forward_scratch_compute + LlamaWeights) has a gfx1201 (RDNA4) kernel compatibility issue. Qwen35 path (forward_scratch_multi) works fine on the same hardware.

6. Daemon serving parity: ⏳ Not tested

HW-003 requires end-to-end daemon serving with committed-token hash matching. Not yet run.

Issues Found

Issue A: gfx1201 × LLaMA forward illegal memory access

LLaMA-family (arch 0/1) forward_scratch_compute crashes on gfx1201 with an illegal memory access during logits download. Qwen35 (arch 5+, DeltaNet) works correctly on the same hardware.

Impact: Blocks all LLaMA-based PP forward testing on gfx1201 (HW-003 dense, plus any arch 0/1 coverage). Likely a missing gfx1201 kernel variant or ISA compatibility gap.

Issue B: 58/100 token divergence between PP=1 and PP=2 (Qwen35)

DeltaNet PP-2 output diverges from single-device output at token 58. Root cause not yet isolated — candidates include FP accumulation differences across stage boundary, synchronization defects, copy-size mismatch, or state-ownership bugs.

Summary

Requirement	Status	Notes
Qwen35 physical weight/state split	✅	Equal per-stage allocation (2.638 GiB/card)
Physical peer access	✅	Bidirectional P2P enabled + verified
Physical stage-boundary execution	⚠️ Partial	Qwen35 forward ran; explicit boundary-copy trace not captured
Qwen35 token parity (PP=1 vs PP=2)	❌	Diverges at token 58; cause unconfirmed
Dense LLaMA physical placement	❌ Blocked	Placement confirmed under emulation only; real forward crashes gfx1201
Dense LLaMA forward parity	❌ Blocked	gfx1201 illegal memory access in `forward_scratch_compute`
Physical unload/reload stability	⏳ Partially tested	100 emulated cycles clean; physical per-GPU teardown not done
PP-only capacity gain	⏳ Not demonstrated	Test model (5.3 GB) fits on one GPU — no PP-required workload tested
Daemon serving parity	⏳ Not tested	Required for HW-003 closure

Remaining work

Fix gfx1201 LLaMA forward crash (Issue A) — the PR blocker for HW-003 dense
Isolate DeltaNet PP divergence cause (Issue B) — synchronization, accumulator, or state-ownership
Run physical per-GPU VRAM teardown test (HW-003 § load/unload criterion)
Run HW-003 manifest gate (store-pp-gate.sh) on physical PP after Issue A fix
Run end-to-end daemon serving with token hash comparison
Retest with the canonical qwen3-0.6b-llama.mq4 artifact (pure LLaMA, arch 0)

Sorry, i dont have ai pro x4...

fivetide · 2026-07-15T08:08:11Z

Thank you—two real R9700s are exactly the validation we needed. The balanced PP allocation and verified P2P result are valuable evidence.

We’re recording the gfx1201 LLaMA illegal-access result as a separate classification blocker; the HF4 artifact and emulated llama_store_pp path do not yet prove a physical PP regression. We’ll reproduce it with the canonical fixture and capture the first failing HIP launch before assigning a cause.

The Qwen35 PP divergence remains a hard blocker for HW-004, not expected numerical drift. After the pending generation-path work, we’ll re-run deterministically with pinned artifacts/prompts and capture boundary, recurrent-state, and copy-order diagnostics.

Thanks again for the thorough environment, artifact, and VRAM data—and no apology needed; a real two-GPU report is precisely what this draft needed.

Replace the hardcoded Q8-shaped KvTierInputs literal (quant_asym*: false, v_mode_bits: 8) with kv_cache.tier_inputs() so asym, FWHT, HFQ8, and all other KV tiers are dispatched correctly. Thread givens_cos/sin from the cache into AttnParams (asym attend kernels unwrap these), flash_partials from ForwardScratch (required for flash attend tiles), and k_scales/v_scales conditionally for HFQ8 (attend.rs panics on None for that tier). Also use gpu.graphs.capture_mode instead of the hardcoded false. Byte-identical no-op for Q8 caches; prevents panics/garbage when the arch-llama modular forward is wired to asym or HFQ8 KV caches. Deferred item from PR warpfront#463. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

9 GPU-free tier-routing tests covering Q8, asym2/3/4, FWHT-asym4, HFQ4, HFQ8, Q4, and F32. Each constructs the KvTierInputs that tier_inputs() now produces for that cache mode and verifies: - uses_givens is true iff rotation buffers are needed (asym/fwht) - write_key and attend_key select the correct paired kernel tier Validates the dispatch path that arch.rs Site A (hipfire-arch-llama) now exercises via kv_cache.tier_inputs() instead of the hardcoded Q8-shaped literal. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…d_dim gates Three changes: 1. llama.rs::KvCache::from_mode: add a clean-error gate for Asym3 alongside the existing Asym4 gate. Both require head_dim=256 (validated on Qwen 3.5); at head_dim=128 the constructors assert and the flash-decode paths are untested. Previously asym3 would panic in new_gpu_asym3_capped; now it returns a user-readable HipError. 2. kv_mode.rs::LLAMA_HFQ_POLICY: expand accepted from [Q8] to [Q8, Asym3, Asym4]. Asym2/Fwht* have no Flat KV constructor and silently fall to Q8 (unimplemented, not broken). Truth-table test updated. 3. carrier.rs::load_bundle: pass ctx.kv_mode_override to resolve() instead of hardcoding "". The override was being silently ignored since the carrier was created — explicit --kv-mode requests had no effect on HFQ llama loads. Result: --kv-mode asym3/asym4 on a head_dim=128 HFQ llama model now gives a clean "requires head_dim=256" error instead of either silent Q8 or a constructor panic. At head_dim=256, asym3/asym4 load and run correctly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Kaden-Schutt · 2026-07-17T05:19:38Z

Carry-forward notes from two PRs being closed in favor of this device-mesh ownership spine:

feat(pp-mtp): Multi-GPU Pipeline Parallelism + MTP speculative decode #382 / narrow replacement Reconcile PP+MTP daemon path from #382 #449 (6d86816f) — PP+MTP: do not inherit the original sprawling historical branch. The reviewable slice was forward_prefill_batch_multi_with_caps, a PP-safe compressed MTP stepper, mtp_mirror tensor-clone helpers, daemon load/generate wiring for pp > 1 plus an explicit compressed .mtp head, and the end-of-generation cycle-depth guard. That slice was validated on hiptrx with a 2-way PP split and emitted spec_path:"pp-mtp". Please carry the capability forward within the unified mesh/generation architecture if it still fits.
fix(qwen35): thread per-device givens into multi-GPU prefill attend #495 — Qwen PP rotated-KV Givens ownership: forward_prefill_chunk must prefer the per-device PrefillBandCtx.givens_cos/sin tensors for FWHT/asym KV under PP rather than reading the global kv_cache.givens_* fields, which are intentionally None in the multi-GPU constructors. The patch is tiny, but the acceptance test needs a real multi-GPU rotated-KV prefill.

Neither stale branch is being merged wholesale into beta; this comment preserves their intent where the new ownership model can absorb it cleanly.

…ont#480)

Gate the QKVZA split-tail route behind HIPFIRE_QKVZA_SPLIT_TAIL=1 and arch_caps.is_rdna3_dgpu(), limiting it to gfx1100/1101/1102. Add a reproducible Qwen3.6-27B A/B prefill benchmark script.\n\nLocal W7900 evidence with scripts/bench_qwen36_qkvza_split_tail_ab.sh, PREFILL=4096, PREFILL_RUNS=3, q8 KV:\n- off median: 589.5 tok/s\n- on median: 613.9 tok/s\n- delta: +4.14% prefill tok/s

Task 4 overreached into pre-checkpoint code. Restore from a32da6b^: GDN/PrefillBandCtx docs, pbs_opts mut params + mask validation, from_ffn cfg(test), five removed expectations, PARO pairs/theta borrows and krot casts (scales cleanup kept), arch_has_wmma + four MQ3 branches, q_dim, duplicate fused_fa3_lloyd_mq4, kv_dim, kv_layer_idx increments, triattn_tap borrow, moe_ffn_dispatch_ep, six unused Lloyd locals. Checkpoint-introduced cleanups stay. Pre-existing diagnostics return. Assisted-by: OpenCode:deepseek/deepseek-v4-flash

…ferences Task 4 quality minor: comments at Qwen35Scratch and the ParoQ4G128 panic diagnostic referenced deleted symbols. moe_ffn_decode -> moe_ffn_decode_impl (graph-capture/allocation claim) and moe_ffn_decode_with_scratch (stream-capture claim); the loader regression hint now names the live ParoBackend -> ParoAugmentor -> load_paro_weight chain. No behavior or error wording change beyond the names. Assisted-by: OpenCode:deepseek/deepseek-v4-flash

…_dtypes Restore commit 6b599c1 reinstated the pre-existing non-cfg dtype projection seam (MoeFfnView::prefill_dtypes -> MoeDtypeSnapshot), returning its dead_code diagnostic in production builds. The method is the projection seam for MoePrefillDtypes::from_ffn (legacy constructor) and the dtype-projection tests, so gate a production-only expect on it with cfg_attr(not(test), ...); from_ffn and test behavior untouched. Verified: qwen35 lib tests 380 passed; all-target clippy no-deps shows no unfulfilled expectation and no prefill_dtypes warning; rustfmt check clean; moe residency + weight-store hybrid boundary scripts pass. Assisted-by: OpenCode:deepseek/deepseek-v4-flash

- c2_reject_unknown_router_dtype: execute validate_frozen_moe_dispatch with a RotationPlan::None, unwhitelisted router dtype (Q8HFQ) and assert the section-6 check_proj rejection names router + dtype (was a no-op fixture). - delta_net_state_abort_checked: drop empty-shell 'ok on empty' test (never called abort_checked; Gpu is not constructible on CPU); strengthen the compile-contract test to assert the empty-state Ok-path precondition and document that the Ok outcome is GPU-bound. - moe_ffn_view_layer_out_of_range_is_result: really exercise moe_ffn_view OOB on a null-tensor Qwen35Weights (was only constructing the error enum); add GPU-ignored frozen_moe_resident_bind_layer_out_of_range twin. Assisted-by: Fixer:deepseek/deepseek-v4-flash

PR #527 slices every expert to inter/tp per rank, so a paged fill must READ the full expert and WRITE a row-gathered subset. Our fill path assumed read length == write length — baked into fetch_batch_into, which read and copied in one step. That assumption, not the gather, is what would have made #527 a painful merge, so it is removed now while the tree is quiet. - Transport::read_batch reads ranges into staging and returns them, with no device involvement. Better layering regardless of TP: the transport no longer needs to know about GPU buffers. - ExpertFillTransform is the disk-bytes -> slot-bytes seam. `None` is identity and keeps today's single-step path, so there is no cost or behaviour change on the current route. - set_fill_transform sizes its scratch once at load, keeping fills free of allocation. Proven without #527: a synthetic EveryNthBlock transform has the same shape as expert_tp_row_gather (full in, 1/tp out) and three tests pin what TP needs — packed_len agrees with real output, ranks partition the expert exactly once, and indivisible geometry fails closed instead of truncating. So read-len != write-len is exercised today rather than discovered later. Until #527 lands, TP + paging is REFUSED at load, and the guard carries its own removal recipe: thread #527's tp_slice into the pool allocator with the packed stride, install a TpRowGather wrapping expert_tp_row_gather, delete the guard. The catalog needs no change either way — it records FULL HFQ ranges and the on-disk layout is TP-independent. #527 requires no changes; the only shared surface is a pub fn it already exports. TP detection is duck-typed on the env knob so this file compiles identically before and after it lands. Neutrality gate PASSES on the identity path; guard verified to fire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G6rH6Y8vMnBpNn2sj3rzqu

nwoolmer · 2026-08-04T10:46:20Z

Integration points with #563 (routed-expert paging) — for later, no action needed here

#563 pages routed experts from the HFQ into bounded per-layer slot pools. Both PRs touch
expert weights, so I checked the overlap and pre-built the seams on our side. #527 needs
no changes — the only shared surface is a pub fn this branch already exports.

Recording it here so whoever merges second has the map.

Already compatible, verified against this branch:

Pointer table — unchanged. ptr_tensor → ptr_slot is a local rename; still
[2·n_exp] F32 with the to_ne_bytes u64 encoding feat(ds4): add DeepSeek-V4-Flash-0731 — routed-expert paging, +28.8% cache budget, and an experimental prefetch adapter #563 pins in a contract test.
Per-rank weights — load_weights_sharded / load_weights_tp each return their own
DeepseekV4Weights, so feat(ds4): add DeepSeek-V4-Flash-0731 — routed-expert paging, +28.8% cache budget, and an experimental prefetch adapter #563's pager (a Mutex field on Weights) is per-rank. No
shared-pager or cross-deviceproblem.
Loader entry — feat(ds4): add DeepSeek-V4-Flash-0731 — routed-expert paging, +28.8% cache budget, and an experimental prefetch adapter #563's paging block lives inside load_weights_inner, so the
tp_slice parameter added here is already in scope at the point it's needed.
select_grouped_lloyd_variant / dispatch_grouped_lloyd → pub(crate) helps:
feat(ds4): add DeepSeek-V4-Flash-0731 — routed-expert paging, +28.8% cache budget, and an experimental prefetch adapter #563 mirrors that selection for its banded prefill and should call it instead (it
already drifted once against beta's i8-WMMA arm).

TP expert slicing — the one real interaction. Under TP a paged fill must READ the full
expert and WRITE a row-gathered subset, so read length ≠ write length. #563 has landed that
seam (Transport::read_batch, ExpertFillTransform, packed-stride pool sizing) and proves
it with a synthetic 1/tp transform, so only the concrete gather is missing. To enable once
this lands:

thread tp_slice into alloc_paged_layer_expert_pool, sizing the pool with the packed
stride (stride / tp) to match what this branch writes to expert_*_stride;
set_fill_transform(Box::new(TpRowGather { .. })) wrapping
weight_store::expert_tp_row_gather;
delete feat(ds4): add DeepSeek-V4-Flash-0731 — routed-expert paging, +28.8% cache budget, and an experimental prefetch adapter #563's TP guard.

The expert catalog needs no change: it records full HFQ ranges, and the on-disk layout
is TP-independent.

EP — the seam is now in too (a2c5a940). #563 has an ExpertOwnership (All | Subset)
and plan_dispatch skips non-owned experts in all three places that matter: the pool
capacity count, the catalog validation, and the resolve loop. So a non-owned expert is never
resolved, never read, and never repointed — it keeps aiming at this branch's zeroed dummy and
contributes zero.

That last point is the whole correctness argument, and it has no runtime error to catch it:
repointing a non-owned expert makes the rank contribute a value the OWNING rank also
contributes, and the all-reduce double-counts. It is pinned by a test that fails with exactly
that message if the filter regresses (verified by neutering the filter — 3 of 5 new tests
fail).

Worth knowing: paging replaces EP's compaction rather than composing with it. This branch
packs owned experts densely via local_of_global; paging assigns slots dynamically. Both
yield base + index·stride, so the pointer-table format is identical — which is why it drops
in rather than fighting the existing scheme.

To enable EP + paging: have alloc_paged_layer_expert_pool build the zeroed dummy and aim
non-owned experts at it (it currently points everything at base), call
set_ownership(Subset(..)) from owns_expert(rank, e), size the pool against n_owned, and
lift #563's EP guard. The guard stays for now only because EP cannot be tested on a single
GPU — the seam is ready, the activation is not claimed.

Known gaps, flagged not fixed:

The safetensors path (upload_layer_routed_experts_from_source) has no paging; the
catalog is built from the HFQ tensor index, so paging is HFQ-only by construction.
load_weights_inner's expert section will need a hand merge — architecturally
compatible, textually overlapping.
feat(ds4): add DeepSeek-V4-Flash-0731 — routed-expert paging, +28.8% cache budget, and an experimental prefetch adapter #563's TP detection is duck-typed on HIPFIRE_TP / HIPFIRE_TP_EXPERT_SLICE, inferred
rather than confirmed against this branch's activation path. Worth a look when retargeting.
The gather would run per fill under paging, not once at load. On a ~4.7 MB expert
that's a strided host copy on a path already read-dominated; TP+paging would be correct
but its cost is unmeasured. A TP-aware layout at quantise time (slice contiguous on disk)
may be the better answer.

One process note: #527 targets master, #563 targets beta, and beta is ~488 commits
ahead. Whichever lands second inherits a larger merge than the file overlap suggests.

Seal MoE lowering and manifest authority across DeepSeek4, MiniMax, and Qwen35. Add deterministic EP/TP parity harnesses, fail-closed topology validation, and complete STEP-002 tracking evidence. Assisted-by: OpenCode:openai/gpt-5.6-sol

Reconciles 488 beta commits with the 363-commit device-mesh branch. Base for both: 5d3683a. 64 conflict events resolved across 155 files (60 content, 2 modify/delete, 2 file-location); 683 files staged. ## Architecture decisions - **Beta is the canonical base for the daemon and shared runtime types.** The merged daemon = upstream/beta's daemon + the device-mesh load-plan (DaemonLoadPlan / daemon_load_plan / pp_bands_from_env) now WIRED into the load handler (admission-first planning, execute_load_plan dispatch). - **LoadedModel layout is beta's flat canonical shape** (arch_id, pp, pp_gpus, ep, kv_cache, dn_state, mtp_*, qwen35_mtp_head, …) plus genuinely-new state only: tp_model, pp_model, vl_image_state. SessionState/PersistState (dual representation) removed; their reset semantics (checkpoint frees with failed-free retention) ported onto LoadedModel::reset_cpu/reset/reset_checked/reset_context. - **superop module + ep.rs restored** (both are beta's; the feature branch had deleted them). ForwardBindings impls adapted to the merged MoeFfnStorage enum; spec_advance is the 6-arg (reset: bool) contract everywhere; all 8 arch impls honor reset via reset_recurrent. - **TS CLI dropped** (beta is Rust-only; crates/hipfire-cli supersedes it). cli/ removed; the Rust control plane is canonical per AGENTS.md. - **TP/dense-PP carriers** (tp_model/pp_model) are wired into unload_model (explicit free()), reset_context (carrier no-op branch), and the daemon's new MeshCarrier generate route (generate_mesh_carrier: render → prefill → host-sampled decode → emit; full-context replay per request). tp/pp parity examples exercise the same carrier API. - **qwen35 hand-coded PP preserved as beta legacy** (Gap 2, advisory- driven): the device-mesh policy table marks qwen35 Pp "Planned" (GEN-001) and load_admitted cannot recreate beta's carrier, so daemon_load_plan classifies qwen35 dense/MoE pp>1 (not VL) into DaemonLoadPlan::LegacyQwen35Pp → load_model_with_kv_backend's restored carrier-registry dispatch (load_qwen35_pp → pp_gpus → generate_multi). load_model_with_kv_backend regained beta's pp>1 registry path incl. the DFlash lm_head/MQ2/MQ3 preflight (extracted into preflight_dflash_draft, shared with load_single_admitted) and the VMM-carrier guard. ## Port debt (follow-ups, not part of this merge) 1. RESOLVED: HIPFIRE_TP / HIPFIRE_EP / HIPFIRE_PP / HIPFIRE_MTP_K env→params routing in crates/hipfire-cli (load_params + apply_draft_max; >1 forwarded only; MTP_K 1..=8; env beats --draft-max for the MTP window; pure parse helpers + unit tests). 2. RESOLVED: daemon_load_plan / admit_path (CAP-001) wired into the daemon load handler (admission precedes unload/GPU work; deferred vs eager lifecycle; ep param; guards extended to ep>1) + MeshCarrier generate branch for tp_model/pp_model. 3. OPEN: #[allow(dead_code)] launch_decode_rotate / launch_gate_side in hipfire-dispatch/src/pipeline/mod.rs — still no callers; cleanup when a caller exists. ## Known - Upstream flake (not merge-caused, crates/hipfire-client byte-identical to beta): load_eligible_then_malformed_clears_and_fails / reset_ack_rejects_missing_and_bad_fields intermittently ETXTBSY on the dummy-daemon spawn under full-workspace parallel load; pass standalone. - hipfire run's generate request was missing attempt_id (daemon requires it since the attempt-correlation protocol); fixed with next_attempt_id(). - daemon example tests did not compile at the merge commit (FinishSummary gained finalized, EmitOutcome gained generation_advanced in a later amend); literals fixed; one beta-era test expectation updated to the merged emitter's stop-sequence quarantine semantics. - PRE-EXISTING (reported, not fixed): qwen3-8b.mq4 + qwen3-8b-dspark.hfq sidecar → RefCell double-borrow panic in hipfire-arch-llama dspark_body.rs load_qwen3_dspark (main_norm pread guard outlives markov load_global_tensor). Single-GPU llama/qwen3 loads with a -dspark sidecar need --spec off until fixed (separate correctness route; out of reconcile scope). - PRE-EXISTING (upstream): redline_daemon_harness.py's prefill capture loop KeyErrors because bench_prefill never emits redline_capture (only bench_decode does) — use --skip-prefill. - ENVIRONMENT (this box): this ROCm 7.2.3 hipcc emits compressed clang offload bundles (CCOB v3, zstd payload); HIP's loader unwraps them, redline-rocr's raw-HSA loader only handles uncompressed __CLANG_OFFLOAD_BUNDLE__ → INVALID_CODE_OBJECT on every cached .hsaco (upstream issue warpfront#569). Worked around for parity validation: officially unbundled (clang-offload-bundler) the exact cached blobs into .hipfire_kernels_plain/ — byte-identical inner gfx1151 ELFs, hash sidecars preserved, selected via HIPFIRE_KERNEL_CACHE. Native CCOB loading in redline-rocr remains unsupported. ROCm split across Nix store paths; ~/.hipfire/rocm-merged symlink root (clr 7.2.3 libamdhip64 + rocm-runtime 7.2.3 libhsa-runtime64) makes the probe reach ROCr. - coherence-gate-dflash.sh is RETIRED per AGENTS.md §0/§3.5 (absent from the tree); correctness route = serve_harness battery/chain + decoded- text inspection (27B) + redline_daemon_harness.py for load/dispatch changes. ## Verification (this session) - cargo check --workspace --examples; cargo test --workspace (exit 0; upstream client flake excluded); clippy --no-deps on hipfire-cli bin, hipfire-loader lib, daemon example; rustfmt scoped (pre-existing non-conformant spots in daemon.rs/main.rs left untouched). - GPU smokes (gfx1151, HIP 7.2): - 9B DFlash canonical pair: τ=5.8571 / accept 0.3905, output "return x + y" — token-identical to pre-Gap baseline (draft md5 590f35403cd7f1d634945233234a12b7). - daemon TP: HIPFIRE_TP=2 HIPFIRE_EMULATE_GPUS=2 hipfire run qwen3-8b.mq4 → admission Tp → TpModel → MeshCarrier, coherent. - daemon dense-PP: HIPFIRE_PP=2 same model → PpModel → MeshCarrier, coherent. - daemon single-GPU: qwen3-8b.mq4 --spec off → 4096d/36L, coherent. - 27B gate (qwen3.5-27b.mq4 + qwen35-27b-dflash-mq4.hfq, draft md5 7b6df2a4ee1c8d933f0a52e187d1860b matches AGENTS.md): - serve_harness battery: 5 turns, 0 attractor / 0 empty, τ 7.5 / 7.28 / 3.25 / 0.97 / 1.68; 3/5 clean stop, 2/5 length-capped (t1 code, t3 factual — t3 also violated the exact-three-sentences constraint with a Thinking-Process preamble); prose τ 0.97 is the documented draft-target prose mismatch, not a regression. (serve-27b-battery.json) - serve_harness chain: 4/5 clean stop, 1/5 length-cap, prefix-cache hits on turns 3-5 (serve-27b-chain.json). - redline_daemon_harness: decode capture STABLE across repeats (hash 993441c8656e5b0c, 947 launches, 21 kernels) — the load+dispatch path is Redline-deterministic. AQL contract probe passed (21 kernels, all loader-kernarg validated); HIP/PM4 parity BIT-EXACT on both backends over the full 947-dispatch sequence vs live-HIP forward (aql_packets exact=True, pm4_ib exact=True; reports .slim/deepwork/evidence-27b/redline-27b-plain-{aql,pm4}.json). Method-qualified: runs used officially unbundled byte-identical inner ELFs in a parallel cache (env bullet); native CCOB loading in redline-rocr remains unsupported (issue warpfront#569). - GitNexus (restored this session): impact analysis on changed symbols (load_params HIGH blast radius, additive-only; dependants match manual grep), detect-changes --scope unstaged: 23 symbols, all expected; daemon example symbols not indexable (example target). - Three deepwork oracle gates (A/B/C) passed; all advisory rounds incorporated.

launch_decode_rotate / launch_gate_side were introduced WITH callers in b1d54d5 (device-mesh MoE refactor) and orphaned by the beta merge: the conflict resolution adopted beta's inline run_moe_decode (activation rotation + gate-side inlined with identical kernel calls), dropping the only call sites. The reconcile kept them under #[allow(dead_code)] as port-debt item 3; this closes it. Verified: zero references repo-wide; GitNexus impact 0 processes / 0 modules for both symbols; cargo check + clippy (lib) on hipfire-dispatch clean of new findings; 223 tests pass (210 ok / 13 ignored) including the run_moe_decode coverage contracts. The gate-side shared_prerot optimization survives via the extracted moe::launch_shared_gate_side / launch_fused_shared_gate helpers, still used by the Step lane.

run_moe_decode_mixed (REAP SP2 Task 3) was never wired: no res.mixed branch exists in run_moe_decode, zero callers, GPU-GATE DEFERRED under #[allow(dead_code)]. Removed it and its private-only dtype_name helper. build_contiguous_permutation + the mixed_dispatch_tests CPU suite are kept (test-covered equivalence invariant; the ignored GPU bucketing stub stays as the owed-validation record).

…parity Merge-consistency fixes found by post-merge revalidation (session 2): - dots-ocr daemon panic: loader (device-mesh) stores dots-ocr state in the ModelState::DotsOcr bundle; daemon (beta) read dead flat fields (dots_ocr_config/dots_ocr_weights/qwen2_state) -> None.unwrap() on any generate. Daemon now reads the bundle via new dots_ocr()/dots_ocr_mut() accessors; resets added at 3 sites; dims/vl flags from the bundle. - deepseek4 daemon panic: deepseek4_pbs flat field dead (moved into Deepseek4Bundle); generate_deepseek4 read it -> panic. Now read from the bundle. Also added missing ModelState::Deepseek4/Minimax dims-match arms. - emulated-ep2-harness compile break vs merged qwen35.rs: Qwen35Scratch init missing ep2_gu_partial0/1, moe_ffn_decode_impl calls missing defer_routed_combine arg, forward_prefill_chunk call missing cfg'd ep2 arg, DeltaNetState::reset_checked -> reset (x2). Verification: qwen35 487 (emulated-ep2-harness) / 425 (plain), loader 106, hipfire-runtime full suite — all green. dots-ocr 1536d load + coherent answer; deepseek4 AR decode coherent; EP2 parity probe tokens identical (max_abs_logit_delta 0.624146 < pinned 0.708164275).

…lently truncated gemv_hfq4g256_moe_down_k8_indexed_batched_expanded (the default routed-down kernel) was specialized to EXACTLY two 256-groups (K=512, the qwen35-A3B routed-down shape) by the 2026-07-11 R17c1 perf rewrite (ec9ec35). For any other K it processed only groups 0-1 and silently dropped the rest: lfm2moe's down K=1792 (7 groups) lost 71% of the dot product, producing deterministic degenerate output (token pinning) at the first MoE layer and compounding through the stack. Fix: guard the 2-group fast path with groups_per_row == 2 (A3B keeps its perf path byte-identical) and restore the generic quads+tail loop for all other group counts, keeping the 4-rows-per-block structure + LAST_COMBINE epilogue. Evidence (gfx1151): - tiny-oracle K=256: MoE-layer cosines 0.989-0.993 -> 0.999+ (the old '4-bit expert noise' gap WAS this bug; OOB group-1 reads) - tiny-oracle real-K (hidden=2048, moe_inter=1792, 32 experts): first MoE layer 0.954 -> 0.995, L4 0.885 -> 0.979 (residual = genuine MQ4 noise) - real LFM2.5-8B-A1B (both the registry fixture and a fresh mq4 re-quant): degenerate pinning -> coherent hash-map explanation with think trace - daemon serve of lfm2.5-8b-a1b.mq4: coherent, banner now shows '2048d 24L 128000 vocab' (added the missing ModelState::Lfm2Moe dims-match arm, same gap previously fixed for ds4/minimax) - qwen3.6-35b-a3b (K=512 fast path): coherent, unchanged - hipfire-runtime suite: 821 passed

- lru_cache_pep8_4nl.txt (md5 6934d440845edbe6118f110b5afdb3a9): 3x four-newline runs over lru_cache_pep8_strict — first A/B round - pep8_40func.txt (md5 7248f5557a7fea19d914fd96e6cb6a22): 40-function PEP-8 code file with 39 triple-newline runs (2500 tokens) — canonical A/B stimulus; normalize flips 39x1358 to 39x271 (token-ID alignment)

STEP-002R + full pivot: checked, exact-retention GPU teardown replaces best-effort frees in construction rollback and unload. Generic core (hipfire-runtime/src/gpu_cleanup.rs): - RetainedGpuTensor / GpuCleanupFailure (tensors + boxed RetryableOwner category, category-preserving retry, redacted Debug) - BundleTeardown trait; retain_free! macro; free_tensor_retained / free_weight_all_checked / free_weight_sidecars_checked / retain_kv_failures - RetryableOwner impl for SingleFreeFailed (frozen stores stay whole) Qwen35 (STEP-002R): - load_bundle -> Qwen35BundleLoadError carrying cleanup owners; checked rollback (KvCache::free_checked, abort_checked x2, free_gpu_checked) - Qwen35Scratch/PrefillBatchScratch abort_checked; construct_kv_cache returns the built KV on reconfig failure - MtpHeadStaging transactional MTP load (MtpHeadLoadError carries retained) - store rollback keeps real dtype/shape (try_free_all_checked); orphaned rejected-replacement buffers thread out of assembly into staging_retained - frozen-fault-inject feature: HIPFIRE_FROZEN_FAIL_STAGE (live env) + HIPFIRE_FROZEN_FAIL_FREE one-shot in free_tensor_checked All arches: - free_checked on every GPU-owning type; BundleTeardown impls for qwen2, qwen35, llama (incl. runtime LlamaWeights/ForwardScratch/PrefillBatchScratch/ dspark), lfm2moe, minimax, cohere2moe, deepseek4, dots-ocr - unload_model dispatches through ModelState::free_checked (exhaustive), retries once before logging; #[must_use] on owner-carrying errors Forensic fixes found while verifying: - dspark_body load_qwen3_dspark held fn-scoped Ref<Vec<u8>> pread guards, panicking ('RefCell already borrowed') on any sidecar load; block-scoped all four guards - free_moe_ffn_checked lacked the packed_expert_owners branch: on RDNA3 packed-MQ4 models it pooled interior expert-view pointers and leaked the owner blobs; now frees sidecars only + blobs once (GPU-tested with a synthetic packing fixture) - free_tensor_checked pooled VMM-owner buffers without arena release (and pooled borrowed views); now arena-aware with checked-retention semantics (VMM teardown verified exact-zero) Tests: GPU fault battery (13), per-arch load/unload VRAM teardown (7 arches + VMM variant), packed-MQ4 teardown, serve_harness battery. CPU suites: runtime 821, loader 106, qwen35 424, qwen2 8, llama 1, lfm2moe 2, minimax 28, cohere2moe 9, deepseek4 117, dots-ocr 27. Residual debt tracked in NEXT-STEPS.md (mid-constructor leaks, PP load rollback).

Terminal teardown surfaces (load_model, unload_model, the qwen35 carrier error path, rollback_unfinished_qwen35, the MTP-head failure logger) used to retry once and then DROP owners that still failed while allocated. They now enqueue into a process-local retained-owner backlog (hipfire_runtime::gpu_cleanup::{enqueue_cleanup_failure, enqueue_retained, retry_backlog, backlog_pending}): whole GpuCleanupFailure entries (both owner categories, never flattened) are retained until the next load_model / unload_model boundary drains them. Owners that still fail stay enqueued (exact-retention) and are reported, so VRAM is reclaimed at the next operation instead of lost until restart. unload_model defers its failure note to the trailing retry_backlog drain: a transient failure this same call reclaims must not report unload as failed with a stale message. No public drop-live-owners API exists (backlog_clear was removed) — the backlog is drained, never silently cleared. HIPFIRE_FROZEN_FAIL_FREE is now continuous-while-set (any non-empty value) instead of one-shot: the initial teardown AND its retry can both be made to fail, which the retained-backlog test needs (the one-shot could never fail a retry, since the retry runs after all injected failures are consumed). The existing cleanup-failure fault test still passes (it clears the env before the retry phase). Test: retained_backlog_enqueues_after_double_failure_and_recovers_on_next_load (GPU): 620 owners enqueued after a double-failed unload, drained by the next load, VRAM recovered. Verified 5/5 runs; per-arch teardown 7/7 + VMM exact, fault battery 14/14, serve_harness 5/5, CPU suites (runtime 821, loader 106, qwen35 424, dots-ocr 27). NEXT-STEPS.md: STEP-002R section marks the terminal log-and-drop gap closed (item 0) alongside the tracked mid-constructor (item 1) and PP-path (item 2) debt.

STEP-002R shipped as PR #18 mirror commits 064e26e + c7f142a (owner-preserving BundleTeardown + retained-owner backlog). GEN-001 flip to complete in b1d54d5 was premature: evidence pending, STEP-003 not complete, PR warpfront#527 and docs/MODELS.md still record Qwen35 PP as partial arch-resident. Refresh Current Status and Execution Priority queue: STEP-003 next, then GEN-002/SPEC-001/VL-001.

…EP-003) Replace the hand-built DeltaNet la_to_device sidecar with placement derived from the state manifest + mesh, defining the sidecar out of existence (device-mesh plan §4 / STEP-001 spec deferred scope). - arch: qwen35_la_devices(cfg, gpus) filters state_manifest Recurrent entries (global-layer keyed) through Gpus::device_for_layer — the compact LA order and owning devices the sidecar duplicated. - qwen35: new_with_quant_multi returns Self only (no Vec<u8>); free_gpu_multi takes &Qwen35Config and derives placement internally. - loader: drop LoadedModel.pp_dn_la_to_device and skeleton_pp param; validate_qwen35_pipeline_layout / reset_qwen35_pipeline_recurrent / validate_reset_layout derive from config + gpus instead of the sidecar; single-model rejection keys off pp_gpus presence. - daemon: reset_qwen35_recurrent, generate_multi context-full + jinja resets, and reset_pp_uncommitted_state derive the map from bundle config + gpus. - examples/tests: update tuple destructures; add GPU-gated test qwen35_la_devices_matches_mesh_placement proving the manifest map equals the mesh and allocations land on owning devices. Loader 106 passed, qwen35 424 passed (21 ignored), daemon example 192 passed, workspace cargo check --all-targets clean. Emulated PP parity gate diverges at step 1 identically with and without this change (pre-existing, HW-004-class).

Increment 1 (9dcb186): manifest-derived state placement, la_to_device sidecar deleted. Increment 2 verification: forward_scratch_layers_multi already runs all DeltaNet recurrent/conv ops through build_delta_net_decode_steps + execute_steps_mesh on the per-device handle. Remaining bespoke ops (QKVZA, FFN, norm) are STEP-004 scope. GEN-001 unblocked (all deps now complete).

…ning forward path Migrate the remaining architecture forward entry points to Step/manifest execution (gates AXIS-001..004). Inventory: .agent-progress/step-004-inventory.md. Qwen35 PP decode (forward_scratch_layers_multi): QKVZA/QKV/gate-up now run through the shared qkvza/qkv/gate_up_via_execute_steps helpers and the KV write + attention ladder through kv_cache_attention_dispatch (new per-device givens override params for asym KV replicas); ~260 lines of bespoke inline dispatch deleted, plus the now-dead run_fused_qkvza_scalar_key and scalar_qkvza_key (+ its obsolete dispatch-pinning test). LFM2 / Cohere2 / MiniMax decode: norms, projections, qk-norm, RoPE, dense FFN, router, and head through Step lists. LFM2 rotation is dtype-driven (dtype_rotation_plan — the plan's Q8-only assumption was wrong for the .mq4 models and produced a token-0 parity break, caught by GPU bisect). Follow-ups (north star): - LFM2/Cohere2 carriers now own the manifest machinery: weight_manifest with ExpertSharded packed-fused surrogates, policy-aware expert_group_manifest (sigmoid_topk + indexed_quantized), state_manifest, and model-owned config-keyed moe_group_plans caches; the routed-MoE expert phases lower through lower_moe_steps + execute_lowered_moe (Single). LFM2's router phase carries ScoreActivation + MoeRoute; Cohere2's is empty (its norm_topk_prob= false top-k stays bespoke, documented exception). - fix(qwen35): DeltaNetState::new_with_quant_multi did not wire the error-feedback residual — the recurrence used the stochastic requantization while the single-GPU ctor wires EF by default, so emulated pp=2 drifted from pp=1 from the first state write (per-layer hidden-state bisect). EF residual now allocated per LA layer on its manifest-derived device. pp_parity cargo test passes 50/50 under HIPFIRE_EMULATE_GPUS=2 (was red at step 1). PHYSICAL PP parity (HW-003/HW-004) is NOT claimed closed — the external token-58 divergence on gfx1201 R9700s remains open pending re-validation. - LFM2's five dead 'mirror' block helpers deleted (user-approved; zero callers crate-wide, stale vs the migrated live path). Parity (pre-migration baselines in .agent-progress/step-004-*-baseline.txt): LFM2 8B-a1b / 350m.mq4 / 350m.q8, Cohere2 North-Mini-Code-1.0, MiniMax M2.7.mq2 — token-identical pre/post migration and pre/post MoE lowering. Workspace check + lib tests green; changed files rustfmt-clean.

…r update - step-004-inventory.md: arch x forward-entry-point table with Step status and justified non-decoder exceptions (vision/MLA/embedding/conv-mixer/interleaved RoPE/per-expert fallback/MiniMax RoPE), follow-up log with resolved items. - step-004-*-baseline.txt / *-migrated.txt: pre/post migration deterministic decode token sequences (LFM2 8B-a1b, 350m.mq4/q8, Cohere2 North-Mini-Code-1.0, MiniMax M2.7.mq2) and the emulated pp=2 baseline. - device-mesh-refactor-tracker.md: STEP-004 marked complete; emulated-PP EF fix recorded with explicit note that HW-003/HW-004 physical gates remain blocked and are not closed by it.

…pass - Repo split: resolved to workspace-boundary-first. warpfront/redline already exists as a public repo while crates/redline{,-dispatch,-rocr} remain in-tree, so the split produced two homes rather than a boundary. - Trainer: located at origin/feat/mtp-dflash-training, 21 ahead / 1832 behind, last touched 2026-06-26, +6747/41 files. This is Path C and it is decaying on the same trajectory that left #527 817 commits behind. - Quantizer competitive claim: deferred to stage 2, post-refactor. - arch/release-and-layering: retired; arch/ksml is the refactor branch. - Parent oracles: hipfire-quantize is the thematic home but is itself 62% main.rs (15,522 of 24,863), so the proposal is a *-quant family rather than a dump. - Naming: recorded that ggml/GGUF are personal initials and the exception rather than the pattern; the objection to ksml is adoption mechanics, not modesty. Name marked PLACEHOLDER.

Ordered plan of record for the saddle re-layering. Companion to the design grounding doc, which holds the measurements. Nineteen items across three tracks: legibility (examples triage, daemon to a real bin, glossary, positioning), ratio (grammar/spec dedup, pflash and quantize eviction, saddle-core extraction, arch slimming), and rescue (Path C trainer, PR #527 disposition). Each item carries a measured size, a risk, its dependencies, and a gate that must pass before it counts as done. Eight ratchets, the headline being the compute:arch line ratio -- 0.70:1 at 8510ca5, 0.85:1 after the ds4 parent eviction, against llama.cpp's 9.7:1. Records four conflicts to resolve rather than paper over, chief among them that AGENTS.md calls PFlash retained legacy research while 2,030 lines of it sit in a production arch crate with 206 daemon references. Explicitly out of scope: rdna-compute, the kernel family, Redline/PM4, radiowave, and the quant formats -- 124,348 lines where the performance advantage actually lives.

E1 (Path C trainer, feat/mtp-dflash-training) is closed: a failed month-1 experiment, out of scope. E2 (#527 disposition) is deferred until the refactor is complete. Drops C2. Harvesting #527's weight_manifest/weight_store/moe_plan was scope imported from that PR: manifests and placement are multi-device *placement*, which is the parallelism concern this refactor is explicitly orthogonal to. saddle-core's contents are therefore grammar, KV, spec orchestration, the capability contract and sampling policy -- and deliberately exclude weight manifests and device placement. Nothing here blocks #527 and #527 blocks nothing here. Records a fifth conflict: with Path C dead, AGENTS.md still lists it as an open investigation (S8), as the roadmap fix for the DDTree gfx1100 regression (S4), and tells 3.6-A3B users to wait for it before using DFlash (S6 pitfall table). The last is actively misdirecting users. Same failure class as the PFlash conflict already recorded.

Tested every remaining lever rather than asserting the conclusion: counting tests consistently on both sides 1.983:1 (compute has MORE test code; neutral) arch-crate weight loading 3,128 lines #527, deferred arch-crate kernel dispatch 674 lines section 6 generic arch code -> saddle-core 3,100 lines makes it worse de-duplicate identical fns 62 lines intentional The tempting one is the third. Of 186 sanctioned generic functions in arch crates, only 13 appear in more than one crate; 173 are single-use. Moving them removes no duplication and turns the substrate into a drawer of unrelated helpers to reclassify lines -- the opposite of legibility. Also documents a real trap found while testing the fifth lever: argmax is duplicated on purpose. The runtime copy has an is_finite() guard because it is the degenerate fallback for sample_top_p; the two arch copies use a bare '>' to match kernels/src/argmax.hip:13 bit-for-bit on the spec-decode path, where disagreeing on +Inf would cause spurious rejections. Both files now say so.

Planning artifact only; authorizes no code changes. Records the measured baseline on 8510ca5 and the finding that motivates the proposal: hipfire's compute-to-architecture line ratio is 0.7:1 where llama.cpp's is 9.7:1. Every generic concern ggml owns once is reimplemented per arch here, which is why DeepSeek V4 is 51,084 lines against llama.cpp's 1,546 for the same model. Also records ~30k lines that are misplaced rather than arch-specific (ds4 parent/ at 20,782 with zero inference-path references, grammar.rs duplicated at 3,935, spec plumbing at 3,370, pflash at 2,030), the correctly-located unsafe surface, the proposed layering with the abstract-the-model-never-the-kernel rule, seven ratchets, and the relationship to PR warpfront#527's completed third.

Kaden-Schutt and others added 20 commits July 16, 2026 19:12

test(minimax): support tokenizer-free prefill fixtures

312c9e9

fix(minimax): gate grouped MoE on supported topology

949d554

fix(minimax): fail closed for unvalidated large batches

fede182

test(mq3): exercise arch-resolved grouped dispatch

9b798a3

test(mq3): use Q8 NRMSE gate for gfx1151 MMQ

e9fefd4

beta: stage PR warpfront#479 MiniMax prefill

e35d2b8

style(dispatch-tests): format KV tier helpers

b907ef4

beta: stage PR warpfront#465 Llama KV dispatch

e66f1a1

chore(deps): refresh workspace dependencies

77dca1d

feat(deps): integrate Rust support crates

b15e6f8

test(deps): add Redline property coverage

9f2b897

test(redline): harden compiled plan invariants

cdd9e53

fix(quantize): remove dead AWQ router exclusion

50a1d5e

docs(beta): record deferred dependency follow-ups

21ab83b

feat(registry): add VibeThinker-3B MQ4/MQ6 (PR warpfront#473)

66cc272

fix(daemon): add dots.ocr text-only generation (PR warpfront#468)

438aa6e

fix(qwen2): apply instruct chat template (PR warpfront#466)

da84d79

This was referenced Jul 17, 2026

feat(pp-mtp): Multi-GPU Pipeline Parallelism + MTP speculative decode #382

Closed

fix(qwen35): thread per-device givens into multi-GPU prefill attend #495

Closed

Bjoern Agent and others added 5 commits July 16, 2026 22:21

refactor(minimax): route projections through execute_steps (PR warpfr…

6090abe

…ont#480)

docs(rdna3): add qkvza split-tail benchmark evidence

53f09cf

docs(rdna3): consolidate qkvza split-tail results

7c1a715

docs(rdna3): record qkvza benchmark system info

3175634

Bjoern Agent added 4 commits August 3, 2026 13:19

fivetide mentioned this pull request Aug 4, 2026

feat(ds4): add DeepSeek-V4-Flash-0731 — routed-expert paging, +28.8% cache budget, and an experimental prefetch adapter #563

Draft

Bjoern Agent added 16 commits August 6, 2026 12:33

refactor(moe): adopt Step and manifest execution

b1d54d5

Seal MoE lowering and manifest authority across DeepSeek4, MiniMax, and Qwen35. Add deterministic EP/TP parity harnesses, fail-closed topology validation, and complete STEP-002 tracking evidence. Assisted-by: OpenCode:openai/gpt-5.6-sol

docs: link STEP-002R tracker entry to PR #18

108be21

docs(device-mesh): record STEP-003 increment 1 evidence

2f3238e

Kaden-Schutt mentioned this pull request Aug 15, 2026

pp=2 bit-exactness gate has been red on gfx1201 since 2026-06-02 (DeltaNet Q8 dither frame counter is topology-dependent) #588

Open

Kaden-Schutt mentioned this pull request Aug 18, 2026

arch: saddle refactor — decompose the monolith into engine/daemon/generate + saddle-core/lab #600

Merged

Navigation Menu

Uh oh!

refactor: unify device mesh, model parallelism, and generation dispatch - #527

refactor: unify device mesh, model parallelism, and generation dispatch#527
fivetide wants to merge 866 commits into
warpfront:masterfrom
fivetide:feature/device-mesh

fivetide commented Jul 12, 2026 •

edited

Loading

Uh oh!

taniguchi-taku-softm commented Jul 14, 2026

Uh oh!

fivetide commented Jul 15, 2026

Uh oh!

Kaden-Schutt commented Jul 17, 2026

Uh oh!

nwoolmer commented Aug 4, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Uh oh!

Conversation

fivetide commented Jul 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Current status

Delivered changes

Architecture support

Cross-cutting generation status

Parallelism implementation

Pipeline parallelism

Tensor parallelism

Expert parallelism

Speculation implementation

Verification

STEP-002 CPU and static checks

STEP-002 hardware and coherence

Earlier branch gates still relevant

Remaining work and known limitations

Hardware validation blockers

Structural gaps

Explicitly deferred STEP-002 debt

Contributor help wanted: AMD multi-GPU validation

DeepSeek4 EP-2 and MiniMax EP-2

Dense PP-2 and Qwen35 PP-2

Dense TP-2

Authoritative work tracker

Completed

Dependency-driven implementation

Axis implementation

Physical validation and final gate

Review guide

Uh oh!

taniguchi-taku-softm commented Jul 14, 2026

Physical PP Validation Findings on 2× R9700 — Partial, Blocked

Environment

Artifact fingerprints

Results

1. VRAM split: pp2_vram_probe ✅ PASS (Qwen35, physical PP-2)

2. Physical peer access ✅ Confirmed

3. Token parity: pp_parity ❌ (58/100 match, Qwen35 PP=1 vs PP=2)

4. Load/unload cycles: pp_unload_reload ✅ PASS (100 cycles, emulated PP)

5. Manifest placement parity: store-pp-gate.sh ❌ Forward failed (emulated PP)

6. Daemon serving parity: ⏳ Not tested

Issues Found

Issue A: gfx1201 × LLaMA forward illegal memory access

Issue B: 58/100 token divergence between PP=1 and PP=2 (Qwen35)

Summary

Remaining work

Uh oh!

fivetide commented Jul 15, 2026

Uh oh!

Kaden-Schutt commented Jul 17, 2026

Uh oh!

nwoolmer commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration points with #563 (routed-expert paging) — for later, no action needed here

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

fivetide commented Jul 12, 2026 •

edited

Loading

1. VRAM split: `pp2_vram_probe` ✅ PASS (Qwen35, physical PP-2)

3. Token parity: `pp_parity` ❌ (58/100 match, Qwen35 PP=1 vs PP=2)

4. Load/unload cycles: `pp_unload_reload` ✅ PASS (100 cycles, emulated PP)

5. Manifest placement parity: `store-pp-gate.sh` ❌ Forward failed (emulated PP)

nwoolmer commented Aug 4, 2026 •

edited

Loading