Repository navigation
Misc. bug: ggml-rpc-server crashes on vulkan when a duplicate copy of libggml-vulkan is dlopened #29138
Description
Name and Version
$ llama-server --version
version: 0.4.1-dev (build 11051, commit 5b59b83)
built with GNU 14.2.0 for Linux x86_64
Operating systems
Linux
Which llama.cpp modules do you know to be affected?
Other (Please specify in the next section)
Command line
ggml-rpc-server --host 0.0.0.0 --port 50052 -cProblem description & steps to reproduce
Starting the RPC server results in an immediate crash.
Since f172be7 ("vulkan: split buffers and debug code into separate files, add shared headers", #28732) the Vulkan backend's internal cross-TU symbols (vk_instance, ggml_vk_init, ggml_vk_instance_init, ...) are exported from libggml-vulkan.so with default visibility (they used to be static).
This interacts badly with ggml_backend_load_all():
1/ ggml_backend_load_best("vulkan", ...) scans the executable directory and dlopens libggml-vulkan.so found there.
2. In a non-GGML_BACKEND_DL build that library has no ggml_backend_score / ggml_backend_init symbols, so load_backend() returns and the handle is immediately dlclosed. This has always happened and used to be harmless.
3. If libggml-vulkan.so in the executable directory is a regular file copy rather than a symlink to the already-loaded libggml-vulkan.so.0 (different inode — e.g. after installing with cp/scp, which dereference symlinks), glibc loads it as a second copy of the library.
4. The duplicate's dynamic relocations bind to the first copy's exported globals (normal ELF interposition). At dlclose, the duplicate's static destructors therefore run against the live library's objects: vk_instance_t::~vk_instance_t() destroys the real vk_instance, leaving device_indices dangling.
Any subsequent Vulkan device use then fails on an "impossible" assert; the exact site varies per run because the vector is dangling, e.g.:
ggml/src/ggml-vulkan/ggml-vulkan.cpp:14735: GGML_ASSERT(device < (int) vk_instance.device_indices.size()) failed
ggml/src/ggml-vulkan/ggml-vulkan.cpp:5173: GGML_ASSERT(idx < vk_instance.device_indices.size()) failed
even though ggml_vulkan: Found 1 Vulkan devices was printed at startup.
First Bad Commit
Relevant log output
Logs
Sep 19 12:13:14 rood ggml-rpc-server[514539]: ggml_vulkan: Found 1 Vulkan devices:
Sep 19 12:13:14 rood ggml-rpc-server[514539]: ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
Sep 19 12:13:14 rood ggml-rpc-server[514539]: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Sep 19 12:13:14 rood ggml-rpc-server[514539]: WARNING: Host ('0.0.0.0') is != '127.0.0.1'
Sep 19 12:13:14 rood ggml-rpc-server[514539]: Never expose the RPC server to an open network!
Sep 19 12:13:14 rood ggml-rpc-server[514539]: This is an experimental feature and is not secure!
Sep 19 12:13:14 rood ggml-rpc-server[514539]: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Sep 19 12:13:14 rood ggml-rpc-server[514539]: Starting RPC server v7.0.0
Sep 19 12:13:14 rood ggml-rpc-server[514539]: endpoint : 0.0.0.0:50052
Sep 19 12:13:14 rood ggml-rpc-server[514539]: local cache : /var/lib/llama-rpc/cache/rpc/
Sep 19 12:13:14 rood ggml-rpc-server[514539]: Devices:
Sep 19 12:13:14 rood ggml-rpc-server[514539]: /home/erik/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:5173: GGML_ASSERT(idx < vk_instance.device_indices.size()) failed
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /usr/local/bin/libggml-base.so.0(+0x187e5) [0x7fe0bbfad7e5]
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /usr/local/bin/libggml-base.so.0(ggml_print_backtrace+0x1df) [0x7fe0bbfadbbf]
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /usr/local/bin/libggml-base.so.0(ggml_abort+0x11e) [0x7fe0bbfadd4e]
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /usr/local/bin/libggml-vulkan.so.0(_Z12ggml_vk_initP23ggml_backend_vk_contextm+0x80b) [0x7fe0b8f6a18b]
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /usr/local/bin/libggml-vulkan.so.0(ggml_backend_vk_init+0x1f9) [0x7fe0b8f6a399]
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /usr/local/bin/libggml-rpc.so.0(ggml_backend_rpc_start_server+0x181) [0x7fe0bbf53b31]
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /usr/local/bin/ggml-rpc-server(+0x75b8) [0x5643c66475b8]
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /lib/x86_64-linux-gnu/libc.so.6(+0x29ca8) [0x7fe0bba35ca8]
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7fe0bba35d65]
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /usr/local/bin/ggml-rpc-server(+0x7df1) [0x5643c6647df1]
Activity
vulkan: hide internal symbols to prevent duplicate-dlopen state destr…
vulkan: hide internal symbols to prevent duplicate-dlopen state destr…
vulkan: hide internal symbols to prevent duplicate-dlopen state destr…
vulkan: hide internal symbols to prevent duplicate-dlopen state destr…
vulkan: hide internal symbols to prevent duplicate-dlopen state destr…