Eval bug: ASSERT failed

[eiffel31]

Name and Version

8133
Vulkan pre-build binary distributed for Linux x86.

Operating systems

Linux

GGML backends

Vulkan

Hardware

Strix Halo

Models

https://huggingface.co/ggml-org/gpt-oss-120b-GGUF

Problem description & steps to reproduce

./llama-b8133_vk/llama-server -m gpt-oss-120b-mxfp4-00001-of-00003.gguf -c 1310720 --host 0.0.0.0 --parallel 10 -kvu

3 request with context length 10000 => OK
4 request with context length 10000 => crash
/home/runner/work/llama.cpp/llama.cpp/ggml/src/ggml-backend.cpp:306: GGML_ASSERT(tensor->data != NULL && "tensor not allocated") failed

(request sent by llama-benchy)

First Bad Commit

No response

Relevant log output

srv params_from_: Chat format: GPT-OSS
srv params_from_: Chat format: GPT-OSS
srv params_from_: Chat format: GPT-OSS
slot get_availabl: id 1 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id 1 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id 1 | task 428 | processing task, is_child = 0
slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id 0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id 0 | task 429 | processing task, is_child = 0
slot get_availabl: id 9 | task -1 | selected slot by LRU, t_last = 19337946462
srv get_availabl: updating prompt cache
srv prompt_save: - saving prompt with length 95, total state size = 6.683 MiB
/home/runner/work/llama.cpp/llama.cpp/ggml/src/ggml-backend.cpp:306: GGML_ASSERT(tensor->data != NULL && "tensor not allocated") failed
/home/user/chat/llama-b8133_vk/libggml-base.so.0(+0x1848b) [0x7f00bca5d48b]
/home/user/chat/llama-b8133_vk/libggml-base.so.0(ggml_print_backtrace+0x21f) [0x7f00bca5d8ef]
/home/user/chat/llama-b8133_vk/libggml-base.so.0(ggml_abort+0x152) [0x7f00bca5dac2]
/home/user/chat/llama-b8133_vk/libggml-base.so.0(ggml_backend_tensor_get+0x109) [0x7f00bca74d59]
/home/user/chat/llama-b8133_vk/libllama.so.0(_ZN21llama_io_write_buffer12write_tensorEPK11ggml_tensormm+0x31) [0x7f00bc0c7dd1]
/home/user/chat/llama-b8133_vk/libllama.so.0(_ZNK14llama_kv_cache16state_write_dataER16llama_io_write_iRKNS_13cell_ranges_tE+0x156) [0x7f00bc0fdae6]
/home/user/chat/llama-b8133_vk/libllama.so.0(_ZNK14llama_kv_cache11state_writeER16llama_io_write_iij+0x295) [0x7f00bc0fdfe5]
/home/user/chat/llama-b8133_vk/libllama.so.0(_ZNK19llama_kv_cache_iswa11state_writeER16llama_io_write_iij+0x2c) [0x7f00bc10fe6c]
/home/user/chat/llama-b8133_vk/libllama.so.0(_ZN13llama_context20state_seq_write_dataER16llama_io_write_iij+0x1a) [0x7f00bc0ba62a]
/home/user/chat/llama-b8133_vk/libllama.so.0(_ZN13llama_context18state_seq_get_dataEiPhmj+0x4d) [0x7f00bc0ba6fd]
./llama-b8133_vk/llama-server(+0x130d70) [0x5606a2f48d70]
./llama-b8133_vk/llama-server(+0x13eb0b) [0x5606a2f56b0b]
./llama-b8133_vk/llama-server(+0x1819ee) [0x5606a2f999ee]
./llama-b8133_vk/llama-server(+0xa177e) [0x5606a2eb977e]
/usr/lib/x86_64-linux-gnu/libc.so.6(+0x29f75) [0x7f00bba33f75]
/usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x87) [0x7f00bba34027]
./llama-b8133_vk/llama-server(+0xa52d5) [0x5606a2ebd2d5]

[adamziel]

Confirming this reproduces on another SWA model, same hardware class. Strix Halo iGPU, Vulkan RADV backend, gemma-4-26B-A4B-it-UD-Q4_K_M.gguf. Tested on llama.cpp builds b8683 and b8693-e8f508269 (the latter built after #21510 merged, which turned out to address a different SWA checkpoint-restore path). Exact same stack trace — ggml-backend.cpp:306 assert in ggml_backend_tensor_get via llama_kv_cache_iswa::state_write, reached from srv prompt_save.

New diagnostic: not really about concurrency, about per-slot context

Mapping --parallel × per-slot -c against the crash:

--parallel	per-slot -c	Result
1	256 K	works — even under 4 queued concurrent requests
4	64 K	works
3	128 K	works
4	128 K	works
4	256 K	crashes on the 2nd sequential warm-up

At 4 × 256 K, the first request returns HTTP 200 cleanly; then srv prompt_save: saving prompt with length N, total state size ≈ 5 MiB fires during the second request's slot assignment and segfaults immediately. No concurrency required — two sequential warm-ups are enough to hit it. The original "crash on 4th concurrent request" symptom is probably the same thing: the 4th concurrent request is the first one hitting a slot that hasn't participated in the first round yet, and state_write on that slot asserts.

Per-slot context size is the threshold variable, not slot count or concurrency depth. Single-slot 256 K is fine (tried 4 queued concurrent requests against --parallel 1 -c 262144 — all serialized cleanly). The break is somewhere between 128 K and 256 K per slot.

Workarounds that don't help

• --kv-unified / -kvu: same crash. Also silently changes -c semantics — n_ctx_seq becomes -c instead of -c / --parallel, which can push per-slot context past n_ctx_train.
• -sps 0 (disable cross-slot prompt similarity): slot selection correctly falls back to LRU, but srv get_availabl → updating prompt cache → prompt_save fires regardless of -sps. Same crash.
• --cache-reuse 0: that's already the default.

Happy to test a proposed fix — reliable repro at --parallel 4 -c 1048576 with Gemma 4 26B-A4B on Vulkan RADV.

[fiesh]

Seems we're also running into this :-(

[fiesh]

This also happens with context size 128k, just much more rarely. It's probably even independent of config size, just becomes harder to trigger with smaller context.

[fiesh]

This problem can now be reproduced very reliably by me. I start (the vulkan-based) llama-server like this:

llama-server --port 8080 --threads-http 4 --model /models/qwen/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf --mmproj /models/qwen/mmproj-Qwen3.6-35B-A3B-F16.gguf --chat-template-kwargs '{"preserve_thinking": true}' --gpu-layers all --cache-ram 32768 --tensor-split 48,52  --temp 0.6 --top_p 0.95 --top_k 20 --min_p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 --ctx-size 1572864 --parallel 6 --host 0.0.0.0

on a system wtih two R9700 AI PROs, and then run parallel_infer.sh, which is:

#!/bin/bash
HOST="http://localhost:8080"

for i in $(seq 1 12); do
  curl -s "$HOST/v1/completions" \
    -H "Content-Type: application/json" \
    -d "{
      \"prompt\": \"Alice has 3 apples. She gives 2 to Bob. Bob then gives half of his apples to Charlie. Charlie gives back to Alice a number of apples equal to the square root of the apples Alice now has. Alice then doubles her apples by buying the same number from the store. Finally, Alice, Bob, and Charlie each eat one apple. How many apples does each person have now? Show your work step by step.\",
      \"max_tokens\": 384,
      \"stream\": false,
      \"temperature\": 0.1
    }" &
done

wait

[fiesh]

@adamziel Please test my PR to see if it also fixes it for you.

[fiesh]

@adamziel Please test my PR to see if it also fixes it for you.

Actually wait, cos now it seems to trigger a different assertion under much harder stress testing...

[fiesh]

@adamziel now please. Ran 512 requests, and all seems good.

[fiesh]

It appears that specifying --kv-unfied also prevents this assertion failure.