Eval bug: DFlash drafter fails to bind (vector::_M_range_check) when target GGUF encodes attention.sliding_window_pattern as an array — Muse-Glimmer-30B official GGUF #26894

Activity

1. CISC commented on Aug 11, 2026

   

   CISC

   on Aug 11, 2026

   Member

   More actions

   Please test linked PR.
2. pcuenca commented on Aug 12, 2026

   

   pcuenca

   on Aug 12, 2026

   Contributor

   More actions

   Hi @bordessoules, I can't reproduce on the current master or 62bf73d. Could you please share a stack trace or full reproducible instructions? I have tested with the following command, where the files are a local download of this repo.

   llama-server -m muse-glimmer-30B-kquant-dynamic.gguf -md dflash-kquant.gguf --spec-type draft-dflash
   
3. bordessoules commented on Aug 13, 2026

   

   bordessoules

   on Aug 13, 2026

   Author

   More actions

   First of all: thank you @pcuenca for taking the time to test, and @CISC — your hunch in the #26900 thread that this issue "may not actually be related" was correct. My original diagnosis was wrong, and I apologize for the noise it caused. I re-investigated from scratch and I believe I found the actual root cause. It has nothing to do with GGUF metadata, on either the target or the drafter side — but there does seem to be a small, reproducible bug underneath, still present on current master. Details below; please tell me if I got anything wrong.

   What I believe the root cause is

   src/llama-model.cpp, default-split layer assignment (line 1352 on current master):

   // default split, by free memory
   splits[i] = free;                    // from ggml_backend_dev_memory()
   ...
   splits[i] /= split_sum;              // free == 0  =>  0/0 = NaN
   const int layer_gpu = std::upper_bound(splits.begin(), splits.begin() + n_devices(),
                                          float(il - i_gpu_start)/act_gpu_layers) - splits.begin();
   auto * dev = devices.at(layer_gpu).dev;

   If every visible device reports 0 bytes of free memory, split_sum == 0 and the normalization turns splits into NaNs. std::upper_bound over NaNs finds no element (x < NaN is false), returns end(), so layer_gpu == n_devices() — and devices.at(layer_gpu) throws exactly the error from my report:

   llama_model_load: error loading model: vector::_M_range_check: __n (which is 1) >= this->size() (which is 1)
   

   (1 >= 1 because I run with a single visible device via CUDA_VISIBLE_DEVICES.)

   Reproduction

   LD_PRELOAD shim that forces ggml_backend_dev_memory to report free = 0 for CUDA devices (real total preserved):

   #define _GNU_SOURCE
   #include <dlfcn.h>
   #include <stddef.h>
   #include <string.h>
   
   typedef void (*real_fn_t)(void *, size_t *, size_t *);
   typedef const char * (*name_fn_t)(void *);
   
   void ggml_backend_dev_memory(void * dev, size_t * free, size_t * total) {
       static real_fn_t real = NULL;
       static name_fn_t name_fn = NULL;
       if (!real) {
           real = (real_fn_t) dlsym(RTLD_NEXT, "ggml_backend_dev_memory");
           name_fn = (name_fn_t) dlsym(RTLD_DEFAULT, "ggml_backend_dev_name");
       }
       real(dev, free, total);
       if (name_fn && strncmp(name_fn(dev), "CUDA", 4) == 0) *free = 0;
   }

   gcc -shared -fPIC -o shim_free0.so shim_free0.c -ldl
   LD_PRELOAD=./shim_free0.so llama-server -m <model> -ngl 99 ...
   

   This reproduces the exact message, word for word, in every combination I tried: b10349 (the build from my original report) and b10380; the Muse Glimmer target with the DFlash drafter, and an unrelated dense model with no speculative decoding at all. So it is not specific to Muse Glimmer or to DFlash.

   It also reproduces naturally, without the shim: saturating the GPU with a separate process (plain cuMemAlloc loop until nothing is allocatable; cuMemGetInfo monitored at 0 free bytes throughout) and then running my original command produces the identical failure sequence from my report, line for line — the ~20 GB target still loads (slowly, apparently falling back to host memory), and then the draft model load crashes:

   llama_model_load: error loading model: vector::_M_range_check: __n (which is 1) >= this->size() (which is 1)
   common_speculative_init_result: failed to load draft model, '.../dflash-kquant.gguf'
   

   That is exactly what I saw on the night of the original report, which explains why the crash hit the draft-model load specifically.

   Why my original report blamed the metadata

   I went back to my logs from that night: the crash happened while loading the draft model, immediately after the ~20 GB target had loaded, at the end of a long benchmarking session with heavy VRAM churn — so the GPU was evidently being held by something else at that moment, matching the natural reproduction above. My metadata "fix" then appeared to work simply because I tested it later, with free VRAM — a timing coincidence that I mistook for causation, and the array-vs-scalar difference between the two conversions made it look convincing.

   To make sure the file itself is innocent, I re-downloaded muse-glimmer-30B-kquant-dynamic.gguf (sha256 matches the Hub LFS oid) and re-ran it on both builds with the exact original command: it loads, binds the DFlash drafter and generates correctly, 10/10 repeated spawns. meta-models/Muse-Glimmer-30B-GGUF is fine as published.

   Possible fix

   Perhaps guard the degenerate case before normalizing, e.g.:

   if (split_sum == 0.0f) {
       // no device reported free memory — fall back to an even split
       // (or fail with a clear out-of-memory message)
       for (size_t i = 0; i < n_devices(); ++i) { splits[i] = float(i + 1) / n_devices(); }
   }

   or clamp layer_gpu to n_devices() - 1 — whichever you prefer. The main value would be that users in this situation get a clear "insufficient device memory" message instead of a cryptic out_of_range, which is what sent me down the wrong path here.

   I'm happy to open a PR for this if useful, and to update this issue's title to reflect the actual bug — something like "model load crashes with vector::_M_range_check when a device reports 0 free memory (NaN in default-split layer assignment)". Or if you prefer to just close this and track it elsewhere, that's fine too. Thanks again for your time.
4. outofcoffee

   added a commit that references this issue on Aug 23, 2026

   docs(examples): add llama.cpp example for Muse-Glimmer-30B (K-Quant-D…

   ...

   c31e09a
5. rrhausen commented on Aug 29, 2026

   

   rrhausen

   on Aug 29, 2026

   More actions

   Independent confirmation of the root cause described above, with a reproduction that needs no LD_PRELOAD shim and no GPU-saturating helper process.

   On a single RTX 4090 (24 GB, sm_89, WSL2) the failure is a pure function of the target model's context size: a large enough -c leaves the device with too little free memory by the time the draft model is loaded.

   llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf -c 130000 -ngl 99 -fa on \
     -ctk q4_0 -ctv q4_0 \
     --spec-type draft-dflash -md Qwen3.8-27B-DFlash2-Q4_K_M.gguf \
     --spec-draft-n-max 5

   -c	result
   8192 / 32000 / 65000 / 100000	loads and runs
   115000 / 125000 / 130000	vector::_M_range_check: __n (which is 1) >= this->size() (which is 1)

   The boundary sits exactly where the headroom runs out: 100000 with --mmproj peaks at 24098 MiB of 24564, i.e. ~466 MiB free, and the draft model is 1.14 GB.

   Ruled out, so it is not the model or the format:

   • Fails at 130000 without --mmproj, and succeeds at 100000 with it — not the mmproj / M-RoPE path.
   • The draft GGUF loads fine standalone on the same binary (it then only fails at dflash requires ctx_other to be set, as expected).

   This matches the NaN-in-default-split explanation exactly, and it is the shape a user actually hits in practice: not a saturated GPU from an unrelated process, but simply asking for a context size that does not leave room for the drafter.

   The practical cost of the current message is that it reads like a model or format incompatibility rather than an out-of-memory condition. I spent a full session concluding DFlash2 was broken for my model before finding this issue — a clear "insufficient free device memory to load the draft model" would have saved all of it. Happy to test a patch.

   Build: 2f3923b, i.e. the branch merged as #27342 / b10f9ca5. CUDA 12.8, CMAKE_CUDA_ARCHITECTURES=89. Target unsloth/Qwen3.8-27B-GGUF (UD-Q4_K_XL), draft z-lab/Qwen3.8-27B-DFlash2-GGUF (Q4_K_M).

   Measurements and write-up produced in a Claude Code session; the hardware and the numbers are my own.

   👍React with 👍1Reacted by mwoigt
6. github-actions

   added

   stale

   on Sep 29, 2026
7. github-actions commented on Oct 13, 2026

   

   github-actionsbot

   on Oct 13, 2026 – with GitHub Actions

   Contributor

   More actions

   This issue was closed because it has been inactive for 14 days since being marked as stale.
8. github-actions

   closed this as not plannedon Oct 13, 2026