Repository navigation
ggml-backend-meta: multi buffers are unsupported leading to vulkan segfault #22197
Description
Activity
gdb backtrace on the debug llama-cli:
Logs
$ gdb --args ./build-dbg/bin/llama-cli -hf unsloth/Llama-3.2-3B-Instruct-GGUF:Q4_K_M -ngl 99 -dev Vulkan0,Vulkan1 -sm tensor -fa 1 GNU gdb (Ubuntu 15.1-1ubuntu1~24.04.1) 15.1 Copyright (C) 2024 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from ./build-dbg/bin/llama-cli... (gdb) r Starting program: /home/matto/llama.cpp/build-dbg/bin/llama-cli -hf unsloth/Llama-3.2-3B-Instruct-GGUF:Q4_K_M -ngl 99 -dev Vulkan0,Vulkan1 -sm tensor -fa 1 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7fffe2bff6c0 (LWP 366916)] [New Thread 0x7fffe23fe6c0 (LWP 366917)] [New Thread 0x7fffe1abc6c0 (LWP 366918)] [New Thread 0x7fffe117a6c0 (LWP 366919)] [New Thread 0x7fffe09796c0 (LWP 366920)] [Thread 0x7fffe09796c0 (LWP 366920) exited] [New Thread 0x7fffe09796c0 (LWP 366921)] [Thread 0x7fffe09796c0 (LWP 366921) exited] [New Thread 0x7fffe09796c0 (LWP 366922)] [Thread 0x7fffe09796c0 (LWP 366922) exited] [New Thread 0x7fffe09796c0 (LWP 366923)] Loading model... |[New Thread 0x7fffcbbff6c0 (LWP 366924)] -[New Thread 0x7fffcb3fe6c0 (LWP 366958)] [Thread 0x7fffcb3fe6c0 (LWP 366958) exited] [New Thread 0x7fffcb3fe6c0 (LWP 366959)] [Thread 0x7fffcb3fe6c0 (LWP 366959) exited] [New Thread 0x7fffcb3fe6c0 (LWP 366960)] [Thread 0x7fffcb3fe6c0 (LWP 366960) exited] [New Thread 0x7fffcb3fe6c0 (LWP 366961)] [Thread 0x7fffcb3fe6c0 (LWP 366961) exited] - Thread 1 "llama-cli" received signal SIGSEGV, Segmentation fault. __gnu_cxx::__atomic_add_dispatch (__val=1, __mem=0x39) at /usr/include/c++/13/ext/atomicity.h:111 111 __atomic_add(__mem, __val); (gdb) bt full #0 __gnu_cxx::__atomic_add_dispatch (__val=1, __mem=0x39) at /usr/include/c++/13/ext/atomicity.h:111 No locals. #1 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_add_ref_copy (this=0x31) at /usr/include/c++/13/bits/shared_ptr_base.h:152 No locals. #2 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count (__r=..., this=<synthetic pointer>) at /usr/include/c++/13/bits/shared_ptr_base.h:1078 No locals. #3 std::__shared_ptr<vk_buffer_struct, (__gnu_cxx::_Lock_policy)2>::__shared_ptr (this=<synthetic pointer>) at /usr/include/c++/13/bits/shared_ptr_base.h:1522 No locals. #4 std::shared_ptr<vk_buffer_struct>::shared_ptr (this=<synthetic pointer>) at /usr/include/c++/13/bits/shared_ptr.h:204 No locals. #5 ggml_vk_tensors_overlap (elementwise=false, b=<optimized out>, a=0x7ffe1f485480) at /home/matto/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:14319 a_buf_ctx = 0x555558eca260 a_buf = <optimized out> b_buf_ctx = 0x555555747e50 b_buf = <optimized out> a_buf_ctx = <optimized out> a_buf = <optimized out> b_buf_ctx = <optimized out> b_buf = <optimized out> a_base = <optimized out> a_size = <optimized out> b_base = <optimized out> b_size = <optimized out> #6 ggml_backend_vk_graph_compute (backend=<optimized out>, cgraph=0x555559bbcc20) at /home/matto/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:14690 src = 0x7ffe1f485480 s = 0 k = 0 dst = 0x7fff899dd160 j = 0 bits = <optimized out> output_nodes = {0x7fff899dd160, 0x0} need_disable = <optimized out> is_topk_moe_single_row = <optimized out> almost_ready = <optimized out> submit = <optimized out> enqueued = <optimized out> op_srcs_fused_elementwise = {false, false, false, false, 37, false, false, false, 42, false, false, false} fusion_string = <optimized out> i = <optimized out> ctx = 0x555555808c30 last_node = <optimized out> first_node_in_batch = <optimized out> submit_node_idx = 0 compute_ctx = std::shared_ptr<vk_context_struct> (use count 2, weak count 6) = {get() = 0x5555598dc5d0} nodes_per_submit = 100 submitted_nodes = <optimized out> submit_count = <optimized out> mul_mat_bytes = 4829184 total_mul_mat_bytes = 4829184 mul_mat_bytes_per_submit = <optimized out> #7 0x00007ffff7e90604 in ggml_backend_meta_graph_compute (backend=<optimized out>, cgraph=<optimized out>) at /usr/include/c++/13/bits/stl_vector.h:1126 bcj = <optimized out> status = <optimized out> j = 0 i = 0 n_backends = <optimized out> backend_ctx = 0x555555af4fa0 needs_rebuild = <optimized out> max_nnodes_raised = <optimized out> iga = 0 ina = 0 get_node_aux = {__backend_ctx = @0x7fffffff2788, __ina = @0x7fffffff2798} set_tmp_data = {__backend_ctx = @0x7fffffff2788} get_cgraph_aux = {__backend_ctx = @0x7fffffff2788, __iga = @0x7fffffff2790} allreduce_fallback = {__n_backends = ./gdb/dwarf2/expr.c:597: internal-error: coerce_pieced_ref: Assertion `closure->pieces.size () == 1' failed. A problem internal to GDB has been detected, further debugging may prove unreliable.
I intend to continue digging into this myself, if anyone could verify reproducibility that'd be great.
@mattngaw I can reproduce on vulkan with Qwen3 4B (full offload) and it segfaulted, somewhere in
ggml_vk_tensors_overlap (elementwise=false, b=0x22c1e16b3a0, a=0x22cac9a9ba0)atllama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp
Line 14495 in 63d93d1
Testing with a single GPU, laptop rtx 4090.
Reacted by Matt Ngaw and issac-cyberThis may be fixed by TheBlueMatt@6e113cb someone needs to clean it up and upstream at some point...
Any chance this will eventually work with AMD GPUs using Vulkan or is explicitly Nvidia-centric?
Would it also work with RPC using mix GPUs (RADv Vulkan + Nvidia Cuda)?
I just tested b9494 and this still crashes with OOM, alloc_tensor_range, with every model (example: Qwen 3.6 27b and 3.5 122b).
"Layer" or "row" work fine.
I'm seeing this as well and finally got some time to look into it. Basically what happens is that when the KV cache gets too big the Vulkan backend allocates multiple buffers to fit it in memory rather than a single buffer, for me that triggers around 1GB or so. If you set
GGML_VK_MEMORY_LOGGER=1and mess with the context size then there's a point where the backend will decide to split up the buffers.I think CUDA is able to allocate large buffers in one go so it doesn't have this problem, but I can't test this. Anyways when the backend uses multiple buffers to store all the data a ggml multi_buffer gets created, which wraps multiple buffers together in a single object (the tensors are still assigned to actual buffers). I think it's done that way so that ggml actually knows that there are several buffers and that they need to be deleted together and all that. The multi_buffer itself doesn't have a valid context and we get our segfault.
#6 0x00007ffff3e7d4ba in ggml_vk_tensors_overlap (a=0x7fffae31b480, b=0x555555d812a0, elementwise=false) at <eve's dir>/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:15620 a_buf_ctx = 0x5555556ef7e0 a_buf = std::shared_ptr<vk_buffer_struct> (use count 2, weak count 0) = {get() = 0x555555ba92c0} b_buf_ctx = 0x5555556edbc0 b_buf = <error reading variable: Cannot access memory at address 0x59> <--- SEGFAULTHere b is a multi buffer. @TheBlueMatt's patch seems to work as it uses the original Vulkan source of the buffer instead of the multi buffer. However it depends on the fact that the tensor has a valid view src (is that always the case?) that links to the original buffer.
If we have a small KV cache and don't have a multi buffer then the meta backend delivers a regular Vulkan buffer (I checked). I'm still looking into it but basically the meta backend was created assuming that there's only one buffer per ggml context.
(gdb) print *(ggml_backend_vk_buffer_context *)b->buffer->context $68 = {device = <error reading variable: Cannot access memory at address 0xa>, dev_buffer = <error reading variable: Cannot access memory at address 0x59>, name = <error: Cannot access memory at address 0x7fff00000001>} (gdb) print *(ggml_backend_vk_buffer_context *)b->view_src->buffer->context $69 = {device = std::weak_ptr<vk_device_struct> (use count 13, weak count 6) = {get() = 0x5555556f0b40}, dev_buffer = std::shared_ptr<vk_buffer_struct> (use count 1, weak count 0) = {get() = 0x5555556ee020}, name = "Vulkan0"}So this explains why it crashes but don't know the code well enough to actually confirm that his patch fixes this properly.
I think I’m hitting the same bug here, once I push
-c/--ctx-sizehigh enough. I'm still figuring out llama.cpp, so I hope this is helpful.Behavior so far:
- --ctx-size 32768: survives
- --ctx-size 64000: segfaults
Hardware:
- OS:
Bluefin 44 / Fedora 44 base - Kernel:
6.19.14-101.fc44.x86_64 - CPU:
AMD Ryzen 7 9800X3D - RAM:
60.4 GiB - GPUs:
AMD Radeon AI PRO R9700(gfx1201),32 GB VRAMAMD Radeon AI PRO R9700(gfx1201),32 GB VRAM
- Vulkan driver:
Mesa RADV llama-server --version:9660 (7dad2f1a1)
I can reproduce it reliably with:
llama-server \ -hf unsloth/Qwen3.6-27B-MTP-GGUF:Q4_K_M \ --host 127.0.0.1 --port 9090 \ --device Vulkan1,Vulkan2 \ -sm tensor -fa on \ --ctx-size 64000 \ --fit off --no-warmup \ --batch-size 4096 --ubatch-size 1024 \ -np 1 \ --spec-type draft-mtp --spec-draft-n-max 2 \ -bs -cb
(I'm pretty sure backend sampling doesn't work with tensor split though, so that can safely be removed).
With this setup, the process exits with 139. The kernel log shows a segfault in libggml-vulkan.so after model load, around slot initialization.
Full gdb trace
warning: File "/var/home/linuxbrew/.linuxbrew/Cellar/gcc/15.2.0_1/lib/gcc/current/libstdc++.so.6.0.34-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load". To enable execution of this file add add-auto-load-safe-path /var/home/linuxbrew/.linuxbrew/Cellar/gcc/15.2.0_1/lib/gcc/current/libstdc++.so.6.0.34-gdb.py line to your configuration file "/var/home/john/.config/gdb/gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/var/home/john/.config/gdb/gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7ffff5fff6c0 (LWP 1322561)] [Thread 0x7ffff5fff6c0 (LWP 1322561) exited] [New Thread 0x7ffff5fff6c0 (LWP 1322562)] [New Thread 0x7fff575ff6c0 (LWP 1322563)] [New Thread 0x7fff569ff6c0 (LWP 1322564)] WARNING: radv is not a conformant Vulkan implementation, testing use only. [New Thread 0x7fff55dff6c0 (LWP 1322565)] [New Thread 0x7fff551ff6c0 (LWP 1322566)] WARNING: radv is not a conformant Vulkan implementation, testing use only. [New Thread 0x7fff47fff6c0 (LWP 1322567)] [New Thread 0x7fff477fe6c0 (LWP 1322568)] [New Thread 0x7ffff52426c0 (LWP 1322569)] [New Thread 0x7fff693ff6c0 (LWP 1322570)] [Thread 0x7ffff52426c0 (LWP 1322569) exited] [Thread 0x7fff693ff6c0 (LWP 1322570) exited] 0.00.495.763 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg) 0.00.495.765 I device_info: 0.00.495.771 I - BLAS : OpenBLAS (0 MiB, 0 MiB free) 0.00.495.913 I - Vulkan0 : AMD Ryzen 7 9800X3D 8-Core Processor (RADV RAPHAEL_MENDOCINO) (32972 MiB, 32536 MiB free) 0.00.495.977 I - Vulkan1 : AMD Radeon AI PRO R9700 (RADV GFX1201) (32624 MiB, 32016 MiB free) 0.00.496.041 I - Vulkan2 : AMD Radeon AI PRO R9700 (RADV GFX1201) (32624 MiB, 32566 MiB free) 0.00.496.046 I - CPU : AMD Ryzen 7 9800X3D 8-Core Processor (61849 MiB, 61849 MiB free) 0.00.496.081 I system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | OPENMP = 1 | REPACK = 1 | 0.00.496.103 I srv init: running without SSL 0.00.496.130 I srv init: using 15 threads for HTTP server 0.00.496.274 I srv start: binding port with default address family [New Thread 0x7fff693ff6c0 (LWP 1322571)] 0.00.496.402 I srv llama_server: loading model 0.00.496.410 I srv load_model: loading model '/var/home/john/.cache/huggingface/hub/models--unsloth--Qwen3.6-27B-MTP-GGUF/snapshots/5cb35eb3dcbf52dbce5f87dbc64df6aaffadcace/Qwen3.6-27B-Q4_K_M.gguf' [New Thread 0x7ffff52426c0 (LWP 1322572)] [New Thread 0x7fff67ec26c0 (LWP 1322573)] [New Thread 0x7fff676c16c0 (LWP 1322574)] [New Thread 0x7fff6672b6c0 (LWP 1322575)] [New Thread 0x7fff65f2a6c0 (LWP 1322576)] [New Thread 0x7fff657296c0 (LWP 1322577)] [New Thread 0x7fff5b16e6c0 (LWP 1322578)] [New Thread 0x7fff5a96d6c0 (LWP 1322579)] [New Thread 0x7fff5a16c6c0 (LWP 1322580)] [New Thread 0x7fff5996b6c0 (LWP 1322581)] [New Thread 0x7fff46ffd6c0 (LWP 1322582)] [New Thread 0x7fff467fc6c0 (LWP 1322583)] [New Thread 0x7fff45ffb6c0 (LWP 1322584)] [New Thread 0x7fff457fa6c0 (LWP 1322585)] [New Thread 0x7fff44ff96c0 (LWP 1322586)] 0.03.666.069 W set_sampler: backend sampling not supported with SPLIT_MODE_TENSOR; using CPU 0.03.666.084 W llama_context: n_ctx_seq (64000) < n_ctx_train (262144) -- the full capacity of the model will not be utilized 0.03.729.665 I srv load_model: creating MTP draft context against the target model '/var/home/john/.cache/huggingface/hub/models--unsloth--Qwen3.6-27B-MTP-GGUF/snapshots/5cb35eb3dcbf52dbce5f87dbc64df6aaffadcace/Qwen3.6-27B-Q4_K_M.gguf' 0.03.729.679 W llama_context: n_ctx_seq (64000) < n_ctx_train (262144) -- the full capacity of the model will not be utilized 0.03.738.369 W load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks 0.03.738.370 W load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024 0.03.738.370 W load_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842 0.03.934.825 I srv load_model: loaded multimodal model, '/var/home/john/.cache/huggingface/hub/models--unsloth--Qwen3.6-27B-MTP-GGUF/snapshots/5cb35eb3dcbf52dbce5f87dbc64df6aaffadcace/mmproj-BF16.gguf' 0.03.934.836 I srv load_model: initializing slots, n_slots = 1 Thread 1 "llama-server" received signal SIGSEGV, Segmentation fault. 0x00007fff695d0d13 in ggml_vk_build_graph(ggml_backend_vk_context*, ggml_cgraph*, int, ggml_tensor*, int, bool, bool, bool)::{lambda(ggml_tensor const*, std::vector<ggml_tensor const*, std::allocator<ggml_tensor const*> > const&)#1}::operator()(ggml_tensor const*, std::vector<ggml_tensor const*, std::allocator<ggml_tensor const*> > const&) const [clone .isra.0] () from /home/linuxbrew/.linuxbrew/Cellar/ggml/0.15.1/libexec/libggml-vulkan.so #0 0x00007fff695d0d13 in ggml_vk_build_graph(ggml_backend_vk_context*, ggml_cgraph*, int, ggml_tensor*, int, bool, bool, bool)::{lambda(ggml_tensor const*, std::vector<ggml_tensor const*, std::allocator<ggml_tensor const*> > const&)#1}::operator()(ggml_tensor const*, std::vector<ggml_tensor const*, std::allocator<ggml_tensor const*> > const&) const [clone .isra.0] () from /home/linuxbrew/.linuxbrew/Cellar/ggml/0.15.1/libexec/libggml-vulkan.so #1 0x00007fff695c747f in ggml_vk_build_graph(ggml_backend_vk_context*, ggml_cgraph*, int, ggml_tensor*, int, bool, bool, bool) [clone .isra.0] () from /home/linuxbrew/.linuxbrew/Cellar/ggml/0.15.1/libexec/libggml-vulkan.so #2 0x00007fff6959c111 in ggml_backend_vk_graph_compute(ggml_backend*, ggml_cgraph*) [clone .lto_priv.0] () from /home/linuxbrew/.linuxbrew/Cellar/ggml/0.15.1/libexec/libggml-vulkan.so #3 0x00007ffff779f5d3 in ggml_backend_meta_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/linuxbrew/.linuxbrew/opt/ggml/lib/libggml-base.so.0 #4 0x00007ffff77950b5 in ggml_backend_sched_graph_compute_async () from /home/linuxbrew/.linuxbrew/opt/ggml/lib/libggml-base.so.0 #5 0x00007ffff68de8c1 in llama_context::graph_compute(ggml_cgraph*, bool) () from /var/home/linuxbrew/.linuxbrew/Cellar/llama.cpp/9660/bin/../lib/../lib/libllama.so.0 #6 0x00007ffff68e154a in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /var/home/linuxbrew/.linuxbrew/Cellar/llama.cpp/9660/bin/../lib/../lib/libllama.so.0 #7 0x00007ffff68e6e2a in llama_context::decode(llama_batch const&) () from /var/home/linuxbrew/.linuxbrew/Cellar/llama.cpp/9660/bin/../lib/../lib/libllama.so.0 #8 0x00007ffff68e862f in llama_decode () from /var/home/linuxbrew/.linuxbrew/Cellar/llama.cpp/9660/bin/../lib/../lib/libllama.so.0 #9 0x00007ffff6e01f11 in common_context_can_seq_rm(llama_context*) () from /var/home/linuxbrew/.linuxbrew/Cellar/llama.cpp/9660/bin/../lib/../lib/libllama-common.so.0 #10 0x00007ffff798436c in server_context_impl::load_model(common_params&) () from /var/home/linuxbrew/.linuxbrew/Cellar/llama.cpp/9660/bin/../lib/libllama-server-impl.so #11 0x00007ffff78def70 in llama_server(int, char**) () from /var/home/linuxbrew/.linuxbrew/Cellar/llama.cpp/9660/bin/../lib/libllama-server-impl.so #12 0x00007ffff7208681 in __libc_start_call_main () from /lib64/libc.so.6 #13 0x00007ffff7208798 in __libc_start_main_impl () from /lib64/libc.so.6 #14 0x00005555555562c5 in _start ()
So this explains why it crashes but don't know the code well enough to actually confirm that his patch fixes this properly.
Okay this patch is only going to work if the tensor in the multi buffer is a view tensor. If it's the original source of the data then it's not going to have a view_src and the patched function isn't going to return the correct Vulkan buffer.
I dug into this and how the multi buffer works is that it basically allocates multiple buffers on the backend when you do a
ggml_backend_alloc, and it returns a multi buffer object instead of a backend buffer. The multi buffer in turn contains an array of pointers to the backend buffers and it has some functions allowing you to say free or clear all the associated buffers in one go. In fact multi buffers were actually created specifically by @0cc4m for the Vulkan backend a long time ago.Since with multi buffers it's possible for tensors to be stored in different physical buffers, during the allocation process each tensor has its associated
tensor->bufferset to identify where it's actually stored. The tensors don't know or care if it's in a multi buffer and when we run the model the backend just goes totensor->bufferto get the actual location of the data.Now the meta backend used for tensor parallel is a virtual backend that picks up the graph like any other backend and internally splits it up so that it runs in parallel across different backend devices. It does a
ggml_backend_allocfor each backend and it keeps track of the returned buffer. That buffer is then passed on to the meta backend context tensors under the assumption that each backend has a single buffer for each context. So in our case with a multi buffertensor->bufferends up pointing to a multi buffer and we get our segfault as the Vulkan backend expects a single Vulkan buffer. I thought of modifying the Vulkan backend to make it support multi buffers but that requires it to know which buffer in the multi buffer is storing that specific tensor, and we don't know that.What I think we can do is:
- Have some sort of way to store the context tensor buffers during
ggml_backend_allocwhen it returns a multi buffer, and later when we init the meta backend transfer those buffers over to the new meta tensors. Obviously we would have to ensure that the meta tensor being initialized is the same as the one that was allocated on the backend in the first place. - Or have the multi buffer object not only point to the buffers it contains but also store the list of tensors that are held inside each buffer.
Of course I might be missing something here and I'm not going to pretend that I understand the meta backend fully as I don't!
Reacted by SleepinDevil- Have some sort of way to store the context tensor buffers during
- added a commit that references this issue
on Jun 26, 2026 - changed the title
[-]Eval bug: segfault under Vulkan --split-mode tensor without explicit -c[/-][+]ggml-backend-meta: multi buffers are unsupported leading to vulkan segfault[/+]on Jul 3, 2026 gdb backtrace on the debug llama-cli:
LogsI intend to continue digging into this myself, if anyone could verify reproducibility that'd be great.
4x rx580
2620v3
ga x99psli
above 4g on (modgrub+setupvar hack)
rebar on (patched by ReBarUEFI)3.6 35b a3b in q4kxl works @ 35tok/s generation
aryan@hope:~/Desktop/AI/best$ systemd-run --user --scope -p MemoryMax=8G -p OOMPolicy=stop -E RADV_PERFTEST=nogttspill ../llama-server -m ./Qwen3.6-27B-UD-Q2_K_XL.gguf -ngl 999 --host 127.0.0.1 --port 8080 --alias qwen35b --tools all -fa on -sm tensor --spec-type draft-mtp --spec-draft-n-max 16 --spec-draft-p-min 0.8 -c 8192
Running as unit: run-p62239-i78712.scope; invocation ID: 25d4e80c6c9d4af1a98f7d8b17e54041
0.00.163.372 W server tools or MCP servers are enabled, using localhost as default CORS origin (change via --cors-origins)
0.00.163.528 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the-lv NCLI arg)
0.00.168.563 W srv llama_server: -----------------
0.00.168.566 W srv llama_server: the following feature(s) are enabled:
0.00.168.567 W srv llama_server: built-in tools (experimental)
0.00.168.567 W srv llama_server: do not expose the server to untrusted environments
0.00.168.567 W srv llama_server: -----------------
0.00.170.749 I srv load_model: loading model './Qwen3.6-27B-UD-Q2_K_XL.gguf'
0.00.992.688 W common_fit_params: failed to fit params to free device memory: llama_params_fit is not implemented for SPLIT_MODE_TENSOR, abort
0.58.568.792 I cmn init: llama threadpool init, n_threads = 6
/home/runner/work/llama.cpp/llama.cpp/ggml/src/ggml-backend-meta.cpp:1156: multi buffers are not supported by the meta backend
[New LWP 62265]
[New LWP 62264]
[New LWP 62263]
[New LWP 62262]
[New LWP 62261]
[New LWP 62260]
[New LWP 62259]
[New LWP 62258]
[New LWP 62257]
[New LWP 62256]
[New LWP 62255]
[New LWP 62254]
[New LWP 62253]
[New LWP 62252]
[New LWP 62251]
[New LWP 62250]
[New LWP 62249]
[New LWP 62248]
[New LWP 62247]
[New LWP 62246]
[New LWP 62245]
[New LWP 62244]This GDB supports auto-downloading debuginfo from the following URLs:
, argc=argc@entry=25, argv=argv@entry=0x7ffd5ffba0f8) at ../sysdeps/nptl/libc_start_call_main.h:59
https://debuginfod.ubuntu.com
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1".
__syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
⚠️ warning: 56 ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S: No such file or directory
#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
56 in ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S
#1 0x000079707c0a061c in __internal_syscall_cancel (a1=, a2=, a3=, a4=, a5=0, a6=0, nr=61) at ./nptl/cancellation.c:49
⚠️ warning: 49 ./nptl/cancellation.c: No such file or directory
#2 __syscall_cancel (a1=, a2=, a3=, a4=, a5=a5@entry=0, a6=a6@entry=0, nr=61) at ./nptl/cancellation.c:75
75 in ./nptl/cancellation.c
#3 0x000079707c11cd4f in __GI___wait4 (pid=, stat_loc=, options=, usage=) at ../sysdeps/unix/sysv/linux/wait4.c:30
⚠️ warning: 30 ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#4 0x000079707c74fcdb in ggml_print_backtrace () from /home/aryan/Desktop/AI/libggml-base.so.0
#5 0x000079707c74fe72 in ggml_abort () from /home/aryan/Desktop/AI/libggml-base.so.0
#6 0x000079707c77e3e4 in ggml_backend_meta_buffer_init_tensor_impl(ggml_backend_meta_simple_tensor_container&, ggml_tensor*) () from /home/aryan/Desktop/AI/libggml-base.so.0
#7 0x000079707c767773 in ggml_gallocr_alloc_graph () from /home/aryan/Desktop/AI/libggml-base.so.0
#8 0x000079707c76de21 in ggml_backend_sched_alloc_graph () from /home/aryan/Desktop/AI/libggml-base.so.0
#9 0x000079707b6f5a7e in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/aryan/Desktop/AI/libllama.so.0
#10 0x000079707b6fce17 in llama_context::decode(llama_batch const&) () from /home/aryan/Desktop/AI/libllama.so.0
#11 0x000079707b6fe1c0 in llama_decode () from /home/aryan/Desktop/AI/libllama.so.0
#12 0x000079707bc45330 in common_init_from_params(common_params&, bool) () from /home/aryan/Desktop/AI/libllama-common.so.0
#13 0x000079707c9c26c3 in server_context_impl::load_model(common_params&) () from /home/aryan/Desktop/AI/libllama-server-impl.so
#14 0x000079707c8ef648 in llama_server(common_params&, int, char**) () from /home/aryan/Desktop/AI/libllama-server-impl.so
#15 0x000079707c8f0ecf in llama_server(int, char**) () from /home/aryan/Desktop/AI/libllama-server-impl.so
#16 0x000079707c02a601 in __libc_start_call_main (main=main@entry=0x57c410044260
⚠️ warning: 59 ../sysdeps/nptl/libc_start_call_main.h: No such file or directory
#17 0x000079707c02a718 in __libc_start_main_impl (main=0x57c410044260 , argc=25, argv=0x7ffd5ffba0f8, init=, fini=, rtld_fini=, stack_end=0x7ffd5ffba0e8) at ../csu/libc-start.c:360
⚠️ warning: 360 ../csu/libc-start.c: No such file or directory
#18 0x000057c410044295 in _start ()
[Inferior 1 (process 62239) detached]
Aborted (core dumped) systemd-run --user --scope -p MemoryMax=8G -p OOMPolicy=stop -E RADV_PERFTEST=nogttspill ../llama-server -m ./Qwen3.6-27B-UD-Q2_K_XL.gguf -ngl 999 --host 127.0.0.1 --port 8080 --alias qwen35b --tools all -fa on -sm tensor --spec-type draft-mtp --spec-draft-n-max 16 --spec-draft-p-min 0.8 -c 8192Check again now that #29266 is merged, the crash should no longer happen for many cases. Performance is still expected to be bad compared to layer split.
Just tested and it's not crashing for me anymore! Going to close this.
Name and Version
version: 8863 (9789512)
built with GNU 13.3.0 for Linux x86_64
Operating systems
Linux
GGML backends
Vulkan
Hardware
Intel(R) Xeon(R) CPU E5-2630L v3
2× AMD Radeon RX 480 (Polaris10, 4 GB each, 8 GB aggregate VRAM)
Linux 6.8.0-110-generic (Ubuntu 24.04)
Mesa 25.2.8 / RADV driver
No peer-to-peer; PCIe 3.0 x16
Models
Llama 3.2 3B (unsloth/Llama-3.2-3B-Instruct-GGUF:Q4_K_M)
Problem description & steps to reproduce
Summary
llama-clisegfaults during model load when using--split-mode tensoron Vulkan unless the user explicitly passes-cwith a "small" context size.Repro
Result:
Segmentation fault (core dumped)after "Loading model..." spinner.Workaround
Any small
-cavoids the crash, this works fine:First Bad Commit
No response
Relevant log output
Logs