Repository navigation
vulkan: allow proxy collectives without external semaphore FD functions - #23
Open
AndrewTilson wants to merge 15 commits into
Open
AndrewTilson wants to merge 15 commits into
AndrewTilson wants to merge 15 commits into
Conversation
…cache views When a per-device allocation exceeds the backend's max buffer size (e.g. a large KV cache), ggml-alloc returns a multi_buffer wrapping several real buffers. Compute-graph views inherited that multi_buffer as their backend buffer, so a backend that casts tensor->buffer->context to its own buffer-context type (the Vulkan backend does, e.g. in ggml_vk_tensors_overlap) dereferenced garbage and crashed with -sm tensor (issue ggml-org#22197). A view aliases its source's storage, so it must reference the source's real sub-buffer: set t_ij->buffer = t_ij->view_src->buffer. This is the correct ggml invariant and a no-op in the single-buffer case. Assisted-by: Claude Opus 4.8
Implements the backend-agnostic comm hook (ggml_backend_comm_init / _allreduce_tensor / _free, discovered by the meta backend via get_proc_address) for the Vulkan backend, so tensor-parallel inference no longer falls back to the meta backend's CPU-barriered butterfly AllReduce. Consumer GPUs have no P2P here, so the reduce stages through host memory, but everything is ordered on the GPU via exported timeline semaphores (no CPU barriers between layers). Each slice is split into chunks: the dedicated transfer queue streams this device's slice out to shared host memory while the compute queue pulls each peer chunk back as soon as it lands, so the two PCIe directions overlap (full-duplex). Partials are cast to F16 before the host transfer to halve the bytes on the bandwidth-bound link and added straight into the fp32 result via the mixed-type add pipeline. Large prefill activations use this pipeline; small (decode) tensors take a single-shot path where the fixed per-call overhead dominates. Roughly 2.5-3x the butterfly fallback; at long context it overtakes -sm layer and is competitive with CUDA/NCCL on prefill. GGML_VK_COMM_OFF disables the custom comm (falls back to butterfly); GGML_VK_COMM_FP32 forces fp32 staging. Assisted-by: Claude Opus 4.8
…duce The GPU-side cross-device ordering imports each peer's OPAQUE_FD timeline semaphore, but OPAQUE_FD payloads are driver-private, so the import only works when all devices share a driver (e.g. two NVIDIA GPUs). On mixed drivers or vendors it is out of spec. Add a portable fallback: a helper thread polls each peer's progress/upload timeline and host-signals a local timeline that the consumer's download is parked on (core timeline semaphores plus host signal/wait, no imported handle). Both the chunked pipeline (prefill) and the single-shot (decode) paths are bridged, so proxy mode no longer drops decode to the meta-backend butterfly. A capability gate (vkGetPhysicalDeviceExternalSemaphoreProperties plus a driverUUID match) selects the proxy deterministically on unsupported configs; GGML_VK_COMM_PROXY forces it, and the import try/catch stays as a safety net. Measured within ~4% of the native-import path on decode and on par for prefill, with byte-identical output. Assisted-by: Claude Opus 4.8
Assisted-by: Claude Opus 4.8
…ackend Assisted-by: Claude Opus 4.8
Assisted-by: Claude Opus 4.8
The ring AllReduce was gated to the native peer-import path (!comm->proxy), so on RADV/cross-vendor setups (which use the CPU-proxy bridge because OPAQUE_FD timeline export is unsupported) GGML_VK_COMM_RING had no effect. Mirror the pipeline's proxy pattern in the ring: the per-step recv wait on the previous neighbour's transfer timeline, and the cross-round WAR wait on the next neighbour's compute timeline, are routed through the device's pxy semaphore and a bridge enqueued for the helper thread. Reserve nsteps+1 pxy values per round (nsteps recv bridges + 1 WAR bridge). Validated byte-identical: native ring == proxy ring == butterfly (660b3d04a269) on 4x A16 forced-proxy. fp32 staging only; F16 is a follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ApKCQ32VLqUW4Kus6tUvBL
…VK_COMM_D2D) Opt-in path that replaces host-memory staging in the tensor-parallel AllReduce with direct peer reads of another GPU's VRAM over PCIe P2P: each device's partial lives in an exportable device buffer (DMA_BUF), peers import it, and the existing comm reads host_buf[k][i] -- now peer VRAM -- unchanged. Targets the O(n^2)-host-bandwidth scaling collapse; pairs with the O(n) ring. Ordering still uses native/CPU-proxy semaphores (proxy auto-selects on RADV), so this is the "D2D data + proxy semaphores" combo. Enables VK_KHR_external_memory_fd + VK_EXT_external_memory_dma_buf. DMA_BUF is the cross-device handle type (OPAQUE_FD memory is spec-locked to one physical device); this matches the amdgpu PCIe-P2P dma-buf mechanism RADV/ROCm use. Status: the fast path is AMD-targeted and UNVALIDATED -- NVIDIA's Vulkan driver rejects cross-device fd import (vkGetMemoryFdPropertiesKHR -> memoryTypeBits=0; confirmed against the Vulkan spec's same-deviceUUID rule and NVIDIA's own statements), and NVIDIA has no fd-import or device-group P2P for unlinked GPUs. So on NVIDIA it logs once and gracefully falls back to host staging -- verified byte-identical and at host speed (pp2048 692 vs 695) on 4x A16, not the slow butterfly. An AMD multi-GPU rig is needed to validate the actual P2P fast path (test recipe accompanies this work, incl. how to prove real VRAM P2P vs a silent amdgpu GTT fallback). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ApKCQ32VLqUW4Kus6tUvBL
The ring transported fp32 chunks; the pipeline already halves host/peer traffic by staging F16. Bring the ring to parity: keep tensors[i] as the fp32 accumulator (matching the pipeline's precision) but cast each chunk to F16 for transport. The cast is folded into the recv step (the just-reduced chunk is the next step's send), so it costs only one extra pre-cast prog value rather than a doubled scheme; per-step up16 slots avoid a send-buffer WAR. GGML_VK_COMM_FP32 still forces fp32. Verified byte-identical greedy output vs fp32 ring / pipeline / butterfly on 4x A16, and no regression (ring-f16 ~= ring-fp32 ~= pipeline on A16 and 4090). The bandwidth win only shows when comm is exposed (comm-bound hosts); on this NVIDIA box the ring's transfer/compute overlap hides the comm, so F16 is neutral here -- same comm-hidden reason the other comm micro-opts are neutral on NVIDIA. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ApKCQ32VLqUW4Kus6tUvBL
Remove all comment lines and trailing comments from the tensor-parallel comm implementation and its device-extension hooks. No behaviour change (verified byte-identical greedy output across ring/proxy/D2D/butterfly after the strip). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ApKCQ32VLqUW4Kus6tUvBL
- Remove the GGML_VK_COMM_D2D peer-buffer path entirely (helpers, VK_KHR_external_memory_fd / VK_EXT_external_memory_dma_buf detection+enable, comm fields, init validation, ensure() branch). On AMD it regresses badly: the dmabuf import lands in GTT (PCIe P2P not established on stock kernels), so peer reads stall the GPU (pp/tg down 40-90%, power collapses). Not viable without box access to validate true VRAM P2P; revisit later. - Make the O(n) ring the default large-tensor AllReduce. The old all-to-all pipeline is now opt-in via GGML_VK_COMM_PIPELINE (was: ring opt-in via GGML_VK_COMM_RING). - Remove the GGML_VK_COMM_OFF toggle (forced the meta-backend butterfly on Vulkan); the custom comm is always better. The generic butterfly fallback in the meta backend stays -- it is the shared fallback for CUDA/SYCL and for Vulkan configs without a usable custom comm (e.g. MoltenVK). Verified byte-identical greedy across ring / pipeline / proxy / fp32 on 4x A16. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ApKCQ32VLqUW4Kus6tUvBL
With the ring as the default there is no reason to keep the slower paths: - Remove ggml_backend_vk_comm_allreduce_pipeline (the O(n^2) all-to-all) and GGML_VK_COMM_PIPELINE. The ring is now the unconditional large-tensor path; the comm->ring flag and pipe_round are gone, and pipeline_ok (the "has two queues" gate the ring needs) is renamed ring_ok. - Remove fp32 staging and GGML_VK_COMM_FP32. The ring always stages F16 (fp32 accumulator preserved); its fp32 branch and use_f16 are removed. Net ~-260 lines. Verified byte-identical greedy output (ring / proxy) and clean build on 4x A16. The decode single-shot and the meta-backend butterfly fallback are untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ApKCQ32VLqUW4Kus6tUvBL
… tree all-reduce Three related changes to the Vulkan -sm tensor (tensor-parallel) all-reduce, developed and measured on 2x Radeon AI PRO R9700 (RDNA4/GFX1201, Mesa 26.1.2): 1. Shrink the comm staging buffers on the prefill->decode transition. ensure() only ever grew the host/tmp buffers, to the peak prefill micro-batch (~10 MB at n_ubatch=512), then reused them for the tiny (~20 KB) decode all-reduces. On RADV the imported external host memory is made visible across devices on every timeline-semaphore signal, at a cost proportional to the resident buffer size, so an oversized leftover cap stalled every decode step (cross-device wait ~770 us vs ~30 us), collapsing decode from ~30 to ~2.4 t/s and staying stuck for the whole session (the AMD "multi-turn crawl"). ensure() now also shrinks when the request is much smaller than cap, with a one-time semaphore wait so the realloc is safe against the previous async all-reduce. Decode after a large prefill: 2.4 -> ~31 t/s, flat across prefill sizes. 2. Decide proxy vs native cross-device sync before creating the progress timelines. They were created as exportable up front, which aborted init on devices that cannot export timeline semaphores (e.g. llvmpipe, RADV on older Mesa) instead of falling back to the portable CPU proxy. Now create exportable timelines only for the native path and plain ones for the proxy path. 3. Add ggml_backend_vk_comm_allreduce_tree: an opt-in recursive halving/doubling all-reduce for power-of-two device counts (2*log2(n) cross-device steps vs the ring's 2*(n-1); bandwidth-optimal). Enabled via GGML_VK_COMM_TREE; the ring stays the default and handles non-power-of-two counts. The step schedule is built as in a reference simulation verified for n=2..8; validated at n=2 (native and forced proxy) to match the ring's greedy output byte-for-byte. Assisted-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SazLJJfgpjt9Kq7JXKKnuw
Prepared with AI assistance for personal-fork review; not an upstream submission.
pwilkin
force-pushed
the
vulkan-tp-p2p
branch
from
September 23, 2026 06:51
13ac901 to
29395e3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Would you be willing to merge this into your branch so if/when you pr makes it to llama repo it incudes these minor changes. As you can see below there was some significant gains from this minor change on my windows system. I am running dual rtx 9700's.
Scope
Allow the existing portable CPU-proxy collective path to initialize when Linux
external semaphore FD functions are unavailable. Host-importable memory remains
required. Missing FD functions select the proxy rather than disabling the fast
collective implementation; the explicit proxy option preserves that selection.
One source file, four additions and three deletions. No harness changes, tuning
switches, diagnostic logging, or experimental Win32 import code are included.
Dependency and attribution
This PR deliberately targets
review/pr25051-base, not main, so it shows onlyour incremental fix to the proposed allreduce implementation:
13ac9015672d4e934d97cd19dd46f3ede4536a41.fix assertion DFlash2 with
--split-mode tensoron CUDA ggml-org/llama.cpp#278583286b033c5dc8bcce9958f724d4e4a93ae589646.The allreduce and DFlash implementations belong to their respective authors.
The tested combined tree also needed port adaptations; those are not part of
this small patch. Do not merge this base into main as if it were our own work.
Historical measurements and limits
Two R9700s, Windows Vulkan, driver 32.0.23033.1002, Q4_K_S target and Q4 DFlash
Combined integration | Passing requests | Aggregate output tokens/s -- | -- | -- Before proxy initialization fix | 36/36 | 33.891 With proxy initialization fix | 36/36 | 38.797 Same-period single-card control | 12/12 | 55.703 Same-period dual-layer control | 12/12 | 44.395drafter, Q8 K/V, flash attention, 32K context, one slot, concurrency one.
Short JSON/code/tool fixtures, same workload across comparisons:
These are historical combined-build results, NOT a hardware benchmark of this
exact isolated patch on its PR25051-only base. Later post-reboot improvements
are not attributed to this fix. Tensor still did not beat single-card.
Short syntax/JSON/tool checks do not establish broad quality or correctness.
Direct TG16 diagnostic (one repetition, different settings from API tests):
17.235 before versus 19.540 tokens/s after; trace changed from fast=0 to
fast=1/proxy=1. No universal speedup is asserted.
Validation
git diff --checkpasses.The fix is backend-level, not Python-harness-specific, but affects only
Scope Allow the existing portable CPU-proxy collective path to initialize when Linux external semaphore FD functions are unavailable. Host-importable memory remains required. Missing FD functions select the proxy rather than disabling the fast collective implementation; the explicit proxy option preserves that selection.consumers using this proposed Vulkan meta-backend collective implementation.
Coordinate with the upstream PR author before any upstream submission.
One source file, four additions and three deletions. No harness changes, tuning
switches, diagnostic logging, or experimental Win32 import code are included.
Dependency and attribution
This PR deliberately targets review/pr25051-base, not main, so it shows only
our incremental fix to the proposed allreduce implementation:
ggml-org#25051
Base: 13ac901.
Historical DFlash integration tests also used
ggml-org#27858
Revision: 3286b03.
Related startup issue: ggml-org#27833.
The allreduce and DFlash implementations belong to their respective authors.
The tested combined tree also needed port adaptations; those are not part of
this small patch. Do not merge this base into main as if it were our own work.
Historical measurements and limits
Two R9700s, Windows Vulkan, driver 32.0.23033.1002, Q4_K_S target and Q4 DFlash
drafter, Q8 K/V, flash attention, 32K context, one slot, concurrency one.
Short JSON/code/tool fixtures, same workload across comparisons:
Combined integration Passing requests Aggregate output tokens/s
Before proxy initialization fix 36/36 33.891
With proxy initialization fix 36/36 38.797
Same-period single-card control 12/12 55.703
Same-period dual-layer control 12/12 44.395
These are historical combined-build results, NOT a hardware benchmark of this
exact isolated patch on its PR25051-only base. Later post-reboot improvements
are not attributed to this fix. Tensor still did not beat single-card.
Short syntax/JSON/tool checks do not establish broad quality or correctness.
Direct TG16 diagnostic (one repetition, different settings from API tests):
17.235 before versus 19.540 tokens/s after; trace changed from fast=0 to
fast=1/proxy=1. No universal speedup is asserted.
Validation
Patch applies to the exact pinned base.
Applied contents match expected extraction; reverse apply check passes.
git diff --check passes.
Clean isolated patch build and matched hardware benchmark.
Regression test for capability/transport selection.
Backend-op correctness and perplexity validation.
Linux native-FD regression and additional Windows workload coverage.
The fix is backend-level, not Python-harness-specific, but affects only
consumers using this proposed Vulkan meta-backend collective implementation.
Coordinate with the upstream PR author before any upstream submission.