Repository navigation
Eval bug: using -devd CUDA0 on draft-dspark causes model to crash #26475
Description
Activity
I get the same problem but only if the draft model is exclusively on its own device. If I add a single layer of the main model onto the device that the draft model is on it runs. However this results in a massive slow down of the output. I have 2x3090 + 96 GB and want to run the main model on 1x3090 + RAM and the dspark on CUDA1 but get the same error.
This fails.
.\llama-server.exe -m "%MODEL%" -ts 24 --n-cpu-moe 38 -fa on --ctx-size 65536 --parallel 1 --batch-size 2048 --ubatch-size 512 --kv-unified --temp 1.0 --top-p 0.95 --min-p 0.00 --metrics --jinja -lv 4 --mmap --fit off --model-draft "%DRAFTM%" --spec-type draft-dspark --spec-draft-n-max 5 -devd CUDA1This works but is very slow because the main model is split across devices.
.\llama-server.exe -m "%MODEL%" -ts 24,1 --n-cpu-moe 38 -fa on --ctx-size 65536 --parallel 1 --batch-size 2048 --ubatch-size 512 --kv-unified --temp 1.0 --top-p 0.95 --min-p 0.00 --metrics --jinja -lv 4 --mmap --fit off --model-draft "%DRAFTM%" --spec-type draft-dspark --spec-draft-n-max 5 -devd CUDA1Following up on #26475 — I hit the same crash class (
GGML_ASSERT(n_graph_inputs < GGML_SCHED_MAX_SPLIT_INPUTS)inggml_backend_sched_split_graph, viaresolve_fused_ops->graph_reserve->sched_reserve), but on an 8-GPU tensor-split setup rather than 2. I tried the workaround from @FlareP1's comment (giving the draft's device a slice of the main model instead of isolating it) plus a few other combinations, and none of them avoided the crash — wanted to share the data points in case it helps narrow down the root cause (possibly related to the graph-node-count instability @ggerganov mentioned in #26369).Environment
- 8x Tesla V100-SXM2-32GB, no NVLink (PCIe only)
- CUDA 11.5 (nvcc), driver 580.173.02
- llama.cpp commit
221f0f6356efe2260023208365705ec5d5a7c8f5(2026-08-02), build:version: 363 (221f0f6), CUDA backend,CMAKE_CUDA_ARCHITECTURES=70 - Target model: DeepSeek-V4-Flash-0731, unsloth UD-Q8_K_XL GGUF (~162GB, 8-shard tensor-split)
- Draft model: DSpark support GGUF converted locally via
convert_hf_to_gguf.py --remote --dsparkfromdeepseek-ai/DeepSeek-V4-Flash-DSpark(verified byte-identical toam17an/DeepseekV4-Flash-20260731-DSpark, so not a conversion issue)
What I tried (all crash identically)
- Baseline: main model split across all 8 GPUs, draft with
--spec-draft-ngl 999and no explicit--spec-draft-device-ngl 999 --tensor-split 1,1,1,1,1,1,1,1 --spec-draft-ngl 999 - Draft pinned to a device that already holds a full share of the main model (mirroring the "shared device" workaround from Eval bug: using -devd CUDA0 on draft-dspark causes model to crash #26475)
-ngl 999 --tensor-split 1,1,1,1,1,1,1,1 --spec-draft-ngl 999 --spec-draft-device CUDA7 - Main model restricted to CUDA1-7 (7-way split), draft isolated on CUDA0, smaller context
-dev CUDA1,CUDA2,CUDA3,CUDA4,CUDA5,CUDA6,CUDA7 -ngl 999 --tensor-split 1,1,1,1,1,1,1 \ --spec-draft-ngl 999 --spec-draft-device CUDA0 -c 65536 - Baseline +
--fit off(to rule out the auto-fit probing pass — this did remove thecommon_fit_params/common_get_device_memory_data_implframes from the backtrace, confirming--fit offskips that pass, but the crash still occurs in the real (non-probing) context init)-ngl 999 --tensor-split 1,1,1,1,1,1,1,1 --spec-draft-ngl 999 --spec-draft-device CUDA0 \ --fit off --batch-size 2048 --ubatch-size 512 --parallel 1 - Small deliberate overlap, mirroring @FlareP1's exact "give the draft's device a small slice" pattern (
-ts 24,1style) scaled to 8 devices-ngl 999 --tensor-split 1,8,8,8,8,8,8,8 --spec-draft-ngl 999 --spec-draft-device CUDA0 \ --fit off --batch-size 2048 --ubatch-size 512 --parallel 1
All five produce the identical stack:
ggml/src/ggml-backend.cpp: GGML_ASSERT(n_graph_inputs < GGML_SCHED_MAX_SPLIT_INPUTS) failed #3 ggml_backend_sched_split_graph () #4 llama_context::graph_reserve(...) #5 llama_context::resolve_fused_ops(...) #6 llama_context::sched_reserve() #7 llama_context::llama_context(...)Takeaway
The device-overlap workaround that helped @FlareP1 on a 2-GPU setup doesn't scale to 8 GPUs — the crash reproduces regardless of overlap ratio or
--fitstate. This is consistent with @ggerganov's note in #26369 about the DSpark graph having a variable node count across batch sizes (397 @ bs=512 vs 417 @ bs=1) — more devices likely means more graph splits, making it easier to exceedGGML_SCHED_MAX_SPLIT_INPUTSeven with the overlap trick. Happy to test patches or provide more logs if useful.Following up on #26475 — I hit the same crash class (
GGML_ASSERT(n_graph_inputs < GGML_SCHED_MAX_SPLIT_INPUTS)inggml_backend_sched_split_graph, viaresolve_fused_ops->graph_reserve->sched_reserve), but on an 8-GPU tensor-split setup rather than 2. I tried the workaround from @FlareP1's comment (giving the draft's device a slice of the main model instead of isolating it) plus a few other combinations, and none of them avoided the crash — wanted to share the data points in case it helps narrow down the root cause (possibly related to the graph-node-count instability @ggerganov mentioned in #26369).Environment
* 8x Tesla V100-SXM2-32GB, no NVLink (PCIe only) * CUDA 11.5 (nvcc), driver 580.173.02 * llama.cpp commit `221f0f6356efe2260023208365705ec5d5a7c8f5` (2026-08-02), build: `version: 363 (221f0f6)`, CUDA backend, `CMAKE_CUDA_ARCHITECTURES=70` * Target model: DeepSeek-V4-Flash-0731, unsloth UD-Q8_K_XL GGUF (~162GB, 8-shard tensor-split) * Draft model: DSpark support GGUF converted locally via `convert_hf_to_gguf.py --remote --dspark` from `deepseek-ai/DeepSeek-V4-Flash-DSpark` (verified byte-identical to `am17an/DeepseekV4-Flash-20260731-DSpark`, so not a conversion issue)What I tried (all crash identically)
1. Baseline: main model split across all 8 GPUs, draft with `--spec-draft-ngl 999` and no explicit `--spec-draft-device` ``` -ngl 999 --tensor-split 1,1,1,1,1,1,1,1 --spec-draft-ngl 999 ``` 2. Draft pinned to a device that **already holds a full share of the main model** (mirroring the "shared device" workaround from [Eval bug: using -devd CUDA0 on draft-dspark causes model to crash #26475](https://github.com/ggml-org/llama.cpp/issues/26475)) ``` -ngl 999 --tensor-split 1,1,1,1,1,1,1,1 --spec-draft-ngl 999 --spec-draft-device CUDA7 ``` 3. Main model restricted to CUDA1-7 (7-way split), draft isolated on CUDA0, smaller context ``` -dev CUDA1,CUDA2,CUDA3,CUDA4,CUDA5,CUDA6,CUDA7 -ngl 999 --tensor-split 1,1,1,1,1,1,1 \ --spec-draft-ngl 999 --spec-draft-device CUDA0 -c 65536 ``` 4. Baseline + `--fit off` (to rule out the auto-fit probing pass — this did remove the `common_fit_params`/`common_get_device_memory_data_impl` frames from the backtrace, confirming `--fit off` skips that pass, but the crash still occurs in the real (non-probing) context init) ``` -ngl 999 --tensor-split 1,1,1,1,1,1,1,1 --spec-draft-ngl 999 --spec-draft-device CUDA0 \ --fit off --batch-size 2048 --ubatch-size 512 --parallel 1 ``` 5. Small deliberate overlap, mirroring [@FlareP1](https://github.com/FlareP1)'s exact "give the draft's device a small slice" pattern (`-ts 24,1` style) scaled to 8 devices ``` -ngl 999 --tensor-split 1,8,8,8,8,8,8,8 --spec-draft-ngl 999 --spec-draft-device CUDA0 \ --fit off --batch-size 2048 --ubatch-size 512 --parallel 1 ```All five produce the identical stack:
ggml/src/ggml-backend.cpp: GGML_ASSERT(n_graph_inputs < GGML_SCHED_MAX_SPLIT_INPUTS) failed #3 ggml_backend_sched_split_graph () #4 llama_context::graph_reserve(...) #5 llama_context::resolve_fused_ops(...) #6 llama_context::sched_reserve() #7 llama_context::llama_context(...)Takeaway
The device-overlap workaround that helped @FlareP1 on a 2-GPU setup doesn't scale to 8 GPUs — the crash reproduces regardless of overlap ratio or
--fitstate. This is consistent with @ggerganov's note in #26369 about the DSpark graph having a variable node count across batch sizes (397 @ bs=512 vs 417 @ bs=1) — more devices likely means more graph splits, making it easier to exceedGGML_SCHED_MAX_SPLIT_INPUTSeven with the overlap trick. Happy to test patches or provide more logs if useful.The workaround for now is as described in #25784 (comment) I believe. You need to compile from source using CMAKE flags
-DGGML_CUDA_PEER_MAX_BATCH_SIZE=128 -DCMAKE_CXX_FLAGS="-DGGML_SCHED_MAX_SPLIT_INPUTS=128". Note that this resolves theGGML_ASSERTspecifically. I have two blackwell GPUs and am able to run when using a build compiled with those flags. I also specify-devd CUDA1without issue, but perhaps there's an issue with 8 GPUs specifically.Reacted by nomonkeynodeal-DGGML_CUDA_PEER_MAX_BATCH_SIZE=128 -DCMAKE_CXX_FLAGS="-DGGML_SCHED_MAX_SPLIT_INPUTS=128"Worked for my 4 GPU split.
Just in case,
GGML_ASSERT(n_graph_inputs < GGML_SCHED_MAX_SPLIT_INPUTS)is a separate problem. Original problem is different and can't be resolved by overriding definesI can reproduce what looks like the same original scheduler/backend issue with Qwen3.6-27B + DFlash on a dual-GPU CUDA setup.
Environment
- llama.cpp commit:
f95de9776(master) - OS: CachyOS / Linux
- CUDA UMD: 13.3
- NVIDIA driver/KMD: 610.43.03
- GPUs: 2× RTX 3060 12 GB
- Target model:
Qwen3.6-27B-IQ4_NL_MTP.gguf - Draft model:
Qwen3.6-27B-DFlash-Q4_K_M.gguf - Speculative mode:
draft-dflash
The target model works normally across both GPUs with MTP.
Reproduction
Relevant parameters:
--model Qwen3.6-27B-IQ4_NL_MTP.gguf --model-draft Qwen3.6-27B-DFlash-Q4_K_M.gguf --spec-type draft-dflash --spec-draft-n-max 15 --device-draft CUDA0 --n-gpu-layers-draft all --split-mode tensor --tensor-split 950,1050The DFlash model is recognized and reaches 100% loading, but context initialization aborts with:
ggml/src/ggml-backend.cpp:898: pre-allocated tensor (output.weight) in a buffer (CUDA1) that cannot run the operation (NONE)The stack follows the same path as the original report:
ggml_backend_sched_backend_id_from_cur() ggml_backend_sched_split_graph() llama_context::graph_reserve() llama_context::sched_reserve() llama_context::llama_context() common_speculative_init_result()I also tried several configurations to rule out a straightforward VRAM or draft-placement issue.
Additional tests
Forcing the entire draft model onto CUDA0:
--device-draft CUDA0 --n-gpu-layers-draft all --override-tensor-draft .*=CUDA0changes the failing tensor, but the same scheduler/backend error remains:
pre-allocated tensor (blk.0.attn_q.weight) in a buffer (CUDA0) that cannot run the operation (NONE)I also reproduced the same class of failure with
--split-mode layerinstead oftensor.With deliberately less balanced layer splits I can trigger ordinary CUDA OOM errors (
cudaMalloc failed: out of memory), which are clearly distinct from this failure. When VRAM is sufficient, initialization instead proceeds until the sameggml_backend_sched_split_graph()/operation (NONE)abort.This appears to be the same original issue reported here, rather than the separate
GGML_SCHED_MAX_SPLIT_INPUTSassert discussed in the comments.Since this reproduces with DFlash/Qwen3.6 rather than DSpARK/DeepSeek, it may indicate that the issue is not specific to DSpARK and can affect speculative draft models with a multi-GPU target more generally.
- llama.cpp commit:
I've got the same issue when attempting to load DeepSeek v4 Flash on 2x gpu (layer split) + ncmoe + dspark loaded exclusively to third small ram gpu.
I'm also having issues with recently released Glimmer (it has dflash drafter)- in split-mode tensor I can fit it on my 2x gpus, but dflash doesnt work until i switch to layer-split.
So, it seems there are some issues with dspark drafters with multiple gpus
This issue was closed because it has been inactive for 14 days since being marked as stale.
Name and Version
latest master
Operating systems
Linux
GGML backends
CUDA
Hardware
nvidia gpus
Models
deepseek flash 0731
Problem description & steps to reproduce
load deepseek with tensor split, then load dspark model with -devd CUDA0
First Bad Commit
No response
Relevant log output