Repository navigation
Eval bug: DFlash speculative decoding fails with --split-mode tensor (shared output.weight in Meta() buffer) #27833
Description
Activity
I can confirm this. My setup is 2x NVIDIA RTX 5060 ti 16GB, Intel® Core™ Ultra 5 245K, 64GB RAM, Ubuntu 24.04.4 LTS
Reacted by Maikel Frias MosqueaSharing a separate tensor-split runtime report:
The shared output.weight / Meta-buffer failure here appears to belong to the DFlash2 initialization problem addressed by #27858.
There is also a separate runtime instability that does not depend on DFlash2:
- Qwen3.8 tensor split reproduces failures with native MTP.
- A related upstream tensor-split failure also reproduces with MTP disabled.
- Two recent runs failed at the same AllReduce slot-reuse synchronization:
ggml_cuda_ar_acquire_slot allreduce.cu:368 cudaEventSynchronize(p->ev_pool[i][slot].ker) - A third fresh tensor-split run terminated during first-request prefill.
- CUDA graphs and graph reuse are not required.
- VRAM exhaustion is not required.
- The failure has no established deterministic token boundary.
- Layer split can also eventually fail, but tensor split is much less stable.
So even if #27858 resolves the DFlash2-specific load/init problem, there appears to be a separate tensor-split runtime stability issue.
Reacted by Michał DąbrowskiI can confirm that this bug exists also for Muse Glimmer UD Q8_K_XL - so it's not just Qwen-related.
- tensor split only (2x 3090) = works
- layer split + dflash = works, but even slower than regular tensor split, so there is no point to use it this way
- tensor split + dflash = crashes with error
D:/a/llama.cpp/llama.cpp/ggml/src/ggml-backend.cpp:941: pre-allocated tensor (output.weight) in a buffer (Meta()) that cannot run the operation (NONE)
Confirming the hypothesis in this report, with the mechanism pinned down: #28777 (comment)
Short version: yes, the draft shares the target's lm_head, but the crash is not caused by the tensor simply "living in the Meta() device". It is caused by a specific split direction:
src/llama-model.cpp:602splitsoutput.weightalong axis 1 (vocab-parallel).handle_mul_matinggml-backend-meta.cppconvertssrc0.axis == AXIS_1intoAXIS_0on the output, so theresult_outputlogits become dim-0 split.- The DFlash2 selector at
src/models/dflash.cpp:494callsggml_top_k(ctx0, res->t_logits, top_k), andGGML_OP_TOP_Kis routed tohandle_per_row, which assertssrc_ss[0].axis != GGML_BACKEND_SPLIT_AXIS_0.
draft-mtp + -sm tensorworks because MTP never callstop_kon the logits in-graph — the direction of the split is what matters, not the sharing itself.Two notes for whoever picks this up:
- Since the head belongs to the target model (the DFlash2 GGUF ships neither
output.weightnortoken_embd.weight, sosrc/models/dflash.cpp:800/:1010borrows it viactx_other), keying a fix offLLM_ARCH_DFLASHdoes not work —ud->modelis the target there. - Forcing the head MIRRORED does remove the assert (verified), but it costs 7-10% decode on the working
draft-mtp + tensorpath (+455 MiB/GPU), so it is not a mergeable fix on its own. Details and numbers in the linked comment. - Also from that link: on my hardware
draft-dflashloses todraft-mtpeven at-sm layer, and would still lose at 26-32% if tensor split were fixed, becauseblock_size=8drafts accept far less (acc 0.24-0.37 vs 0.48-0.81). Worth knowing before prioritising this.
Name and Version
version: 0.3.0-dev (build 10661, commit 3217633)
built with Clang 20.1.8 for Windows x86_64
Operating systems
Windows
GGML backends
CUDA
Hardware
2x NVIDIA RTX 3060 12GB, Intel Xeon E5-2690 v4, 64GB RAM
Models
Target: unsloth/Qwen3.8-27B-GGUF (UD-Q4_K_M)
Draft: z-lab/Qwen3.8-27B-DFlash2-GGUF (Q4_K_M)
Problem description & steps to reproduce
DFlash speculative decoding (--spec-type draft-dflash) fails to load when --split-mode tensor is used with multiple CUDA devices. The same setup works correctly with --spec-type draft-mtp, and DFlash itself works when switching to --split-mode layer, so the issue appears specific to the interaction between DFlash and tensor split.
The DFlash draft model shares the target model's output.weight (per the auto-setup that reuses the target's embedding + lm_head). Under tensor split that tensor lives in the combined Meta() device spanning both GPUs, and the draft context cannot operate on it.
There is also a second, likely related failure: with tensor split and DFlash, the server crashes earlier during warmup with GGML_ASSERT(obj_new) failed at ggml.c:1804. Passing --no-warmup gets past that point and surfaces the output.weight error.
Steps to reproduce:
Run with tensor split -> crashes during warmup:
llama-server.exe --model Qwen3.8-27B-UD-Q4_K_M.gguf --model-draft Qwen3.8-27B-DFlash2-Q4_K_M.gguf --device "CUDA0,CUDA1" --spec-draft-device CUDA1 --gpu-layers all --split-mode tensor --tensor-split "1,1" --fit off --ctx-size 8192 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --spec-type draft-dflash --spec-draft-n-max 7 --parallel 1 --jinja
Same command plus --no-warmup -> fails when loading the draft model.
Same command with --split-mode layer instead of tensor -> loads and generates correctly.
Same command with --spec-type draft-mtp and --split-mode tensor -> loads and generates correctly (~39 t/s).
Reducing --ctx-size from 122880 to 8192 does not change either failure.
Performance comparison on this hardware, same prompt:
Because DFlash cannot currently be combined with tensor split, users on multi-GPU setups have to trade away the tensor-split speedup to use it, which on this hardware makes DFlash a net loss despite reasonable draft acceptance.
First Bad Commit
No response
Relevant log output
Case 1: --split-mode tensor, crash during warmup
llama_prepare_model_devices: creating a Meta device with 2 devices
llama_prepare_model_devices: - device 0: CUDA0
llama_prepare_model_devices: - device 1: CUDA1
llama_prepare_model_devices: using device Meta() (Meta()) (unknown id) - 22510 MiB free
load_tensors: offloaded 66/66 layers to GPU
load_tensors: CUDA_Host model buffer size = 682.03 MiB
load_tensors: Meta() model buffer size = 7338.51 MiB
llama_context: n_ctx = 8192
sched_reserve: graph nodes = 6151
sched_reserve: graph splits = 2
cmn init: llama threadpool init, n_threads = 14
cmn common_init_: warming up the model with an empty run - please wait ... (--no-warmup to disable)
D:/a/llama.cpp/llama.cpp/ggml/src/ggml.c:1804: GGML_ASSERT(obj_new) failed
Case 2: --split-mode tensor + --no-warmup, fails loading the draft model
cmn init: llama threadpool init, n_threads = 14
common_speculative_init_result: loading draft model 'Qwen3.8-27B-DFlash2-Q4_K_M.gguf'
D:/a/llama.cpp/llama.cpp/ggml/src/ggml-backend.cpp:932: pre-allocated tensor (output.weight) in a buffer (Meta()) that cannot run the operation (NONE)
Case 3: --split-mode layer, works
common_speculative_init_result: loading draft model 'Qwen3.8-27B-DFlash2-Q4_K_M.gguf'
srv load_model: initializing, n_slots = 1, n_ctx_slot = 8192, kv_unified = 'false'
common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash'
common_speculative_impl_draft_dflash: - n_max=2, n_min=0, p_min=0.00
common_speculative_impl_draft_dflash: - block_size=8, mask_token_id=248070, n_extract=5, sample_from_anchor=true
srv llama_server: model loaded
slot print_timing: id 0 | task 0 | prompt eval time = 4867.92 ms / 1588 tokens (3.07 ms per token, 326.22 tokens per second)
slot print_timing: id 0 | task 0 | eval time = 14974.27 ms / 343 tokens (43.78 ms per token, 22.84 tokens per second)
slot print_timing: id 0 | task 0 | draft acceptance = 0.53939 (178 accepted / 330 generated), mean len = 2.08
Case 4: --split-mode tensor with draft-mtp, works
common_speculative_init_result: creating MTP draft context against the target model 'Qwen3.8-27B-UD-Q4_K_M.gguf'
spec common_specu: adding speculative implementation 'draft-mtp'
spec common_specu: - n_max=2, n_min=0, p_min=0.00, n_embd=5120, backend_sampling=1
W set_sampler: backend sampling not supported with SPLIT_MODE_TENSOR; using CPU
srv llama_server: model loaded
slot print_timing: id 0 | task 0 | n_gen = 3226, tg = 39.41 t/s