Misc. bug: Inconsistent Vulkan segfault

[RobbyCBennett]

Name and Version

library 531cb1c (gguf-v0.4.0-2819-g531cb1c2)

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

No response

Problem description & steps to reproduce

1. Compile the program below
2. Run it a thousand times and it will probably have a segmentation fault at least once. I used the gdb debugger.

Simple program:

#include "llama.h"

static void handleLog(enum ggml_log_level level, const char *text, void *user_data) {}

int main(int argc, char **argv)
{
  llama_log_set(handleLog, 0);

  char path[] = "/your-path-to/llama.cpp/models/ggml-vocab-llama-bpe.gguf";
  struct llama_model_params params = llama_model_default_params();
  struct llama_model *model = llama_load_model_from_file(path, params);
  llama_free_model(model);

  return 0;
}

Shell script to run the program several times:

#! /bin/sh

PROGRAM=llama-bug
LOG=debug.log
COUNT=1000

rm -f "$LOG"

for i in `seq 1 $COUNT`; do
	gdb -batch -ex run -ex bt "$PROGRAM" >> "$LOG" 2>> "$LOG"
done

First Bad Commit

No response

Relevant log output

GDB output from crash caused by /lib/x86_64-linux-gnu/libnvidia-eglcore.so.535.183.01

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
ggml_vulkan: Compiling shaders..............................Done!

Thread 3 "[vkrt] Analysis" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe35a8640 (LWP 1789333)]
0x00007fffeff1cb00 in ?? () from /lib/x86_64-linux-gnu/libnvidia-eglcore.so.535.183.01
#0  0x00007fffeff1cb00 in ?? () from /lib/x86_64-linux-gnu/libnvidia-eglcore.so.535.183.01
#1  0x00007ffff0246f1d in ?? () from /lib/x86_64-linux-gnu/libnvidia-eglcore.so.535.183.01
#2  0x00007fffeff1fcfa in ?? () from /lib/x86_64-linux-gnu/libnvidia-eglcore.so.535.183.01
#3  0x00007ffff7a1dac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#4  0x00007ffff7aaf850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

GDB output from crash with unknown cause

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
ggml_vulkan: Compiling shaders..............................Done!

Thread 3 "[vkrt] Analysis" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe35a8640 (LWP 1750868)]
0x00007fffeff1cb00 in ?? ()
#0  0x00007fffeff1cb00 in ?? ()
#1  0x000000006746139a in ?? ()
#2  0x0000000002a1b0d8 in ?? ()
#3  0x0000000067461399 in ?? ()
#4  0x00000000000e6817 in ?? ()
#5  0x00005555561076c0 in ?? ()
#6  0x00007fffeff1ef10 in ?? ()
#7  0x0000000000000000 in ?? ()

[jeffbolznv]

This might be a driver bug. Can you try the latest drivers?

I think there's also a chance this could be caused by the ggml-vulkan backend not destroying the VkDevice/VkInstance before the process is terminated. That's something we should look into fixing.

[slaren]

I think there's also a chance this could be caused by the ggml-vulkan backend not destroying the VkDevice/VkInstance before the process is terminated.

We may need to add a function to destroy a backend and release all the resources, otherwise calling ggml_backend_unload to unload a dynamically loaded backend may result in a leak.

[jeffbolznv]

IMO issue #10420 is also a question about the object model for ggml backends, i.e. should it be possible for each thread to have its own VkInstance/VkDevice and what ggml/llama object should their lifetime be tied to.

[0cc4m]

This might be a driver bug. Can you try the latest drivers?

I think there's also a chance this could be caused by the ggml-vulkan backend not destroying the VkDevice/VkInstance before the process is terminated. That's something we should look into fixing.

I think this is a bug I often observe on Linux, but only on Nvidia. It happens when exiting the application, so some issue with clean-up. I haven't looked into it yet.

IMO issue #10420 is also a question about the object model for ggml backends, i.e. should it be possible for each thread to have its own VkInstance/VkDevice and what ggml/llama object should their lifetime be tied to.

I've tried building the global stuff in the way I think CUDA is handling it in the background, but it's not done yet. It should be possible to keep the device and instance global if all temporary variables stay attached to the backend instance. Command buffers are probably not the only thing where that hasn't been implemented yet.

[0cc4m]

This is how the crash looks for me. It happens in some Nvidia driver thread after all the ggml code has already exited:

Thread 7 "[vkps] Update" received signal SIGSEGV, Segmentation fault.

This is the thread:

* 7    Thread 0x7fffd56006c0 (LWP 683442) "[vkps] Update"   0x00007fffe5401960 in ?? () from /lib/x86_64-linux-gnu/libnvidia-eglcore.so.565.57.01

This is the stack trace:

#0  0x00007fffe5401960 in ?? () from /lib/x86_64-linux-gnu/libnvidia-eglcore.so.565.57.01
#1  0x00007fffe57392b4 in ?? () from /lib/x86_64-linux-gnu/libnvidia-eglcore.so.565.57.01
#2  0x00007fffe5404dfa in ?? () from /lib/x86_64-linux-gnu/libnvidia-eglcore.so.565.57.01
#3  0x00007ffff729ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#4  0x00007ffff7329c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

@jeffbolznv Do you happen to know what the [vkps] Update thread is? I don't know what is/isn't getting cleaned up in a way to cause the Nvidia driver to segfault. No other driver shows this issue.

Interestingly @RobbyCBennett saw it in one of the other Nvidia threads ([vkrt] Analysis).

Resolving this has gotten a little more important since @LostRuins reported that a crash on exit on Windows in certain cases causes a system crash (BSOD). Might be the same cause.

[0cc4m]

I hacked together a commit (335f48a) where the devices and Vulkan instance get cleaned up properly (at least I think so, validation layers didn't print anything), but the Nvidia driver still segfaults.

[jeffbolznv]

They're both threads created by the driver. I'm pretty sure they should get shut down when the VkDevice is destroyed.

I hacked together a commit (335f48a) where the devices and Vulkan instance get cleaned up properly

Is this relying on the static destructor for the vk_instance pointer? I think that may happen too late. Is there a hook where we can destroy the objects before the process is terminated?

[slaren]

Is there a hook where we can destroy the objects before the process is terminated?

Not at the moment. I may add destructors for the backend_device and backend_reg objects in the future, but these would still rely on a static destructor to be called normally when exiting the application. I understand that static destructors can be risky due to the order of destruction, but I am not sure why that should be a problem for the Vulkan driver. I would very much prefer to avoid adding a function to shutdown ggml unless it is absolutely necessary.

[RobbyCBennett]

I had some system problems when updating the driver, but I finally got some results. I still see segmentation faults with the new driver. I haven't tried the commit 335f48a yet.

The different types of seg faults I got:

• Thread 5 "[vkps] Update" received signal SIGSEGV, Segmentation fault.
• Thread 3 "[vkrt] Analysis" received signal SIGSEGV, Segmentation fault.
• Thread 5 received signal SIGSEGV, Segmentation fault.

Here's some more system information if that helps at all:

• Ubuntu 22 before and Ubuntu 24 now
• NVIDIA RTX 4090

[0cc4m]

I would very much prefer to avoid adding a function to shutdown ggml unless it is absolutely necessary.

It's probably not absolutely necessary, since this issue only appears on Nvidia. Their driver should handle this gracefully.

[jeffbolznv]

@0cc4m or @RobbyCBennett, can you try adding a call into ggml-vulkan to destroy the VkDevice and VkInstance right before dlclose is called in unload_backend?

I may add destructors for the backend_device and backend_reg objects in the future, but these would still rely on a static destructor to be called normally when exiting the application.

I suspect (not sure) it's OK to invoke the cleanup from a static destructor in the main executable (or ggml.so?), as long as it's before ggml-vulkan or the vulkan driver libraries have been unloaded.

Linux doesn't give a good way to have this entirely self-contained in the vulkan driver or in ggml-vulkan. I think some kind of call from ggml is needed.

[RobbyCBennett]

I don't know this code and I haven't made any commits to this project. I don't see any VkDevice or VkInstance types. Maybe @0cc4m can make this change.

[RobbyCBennett]

Here's an idea for a potential workaround: provide a way to have a preferred backend. For my example, if CUDA is available, then use CUDA otherwise use Vulkan. I don't currently see a way to specify the preferred backend. Maybe it could look like the following.

enum llama_specific_backend_type {
    LLAMA_SPECIFIC_BACKEND_TYPE_CUDA,
    LLAMA_SPECIFIC_BACKEND_TYPE_VULKAN,
    // others...
};

const llama_specific_backend_type PREFERRED_BACKENDS[] = {
    LLAMA_SPECIFIC_BACKEND_TYPE_CUDA,
    LLAMA_SPECIFIC_BACKEND_TYPE_VULKAN,
};

int main()
{
  llama_set_backend(PREFERRED_BACKENDS, sizeof(PREFERRED_BACKENDS) / sizeof(PREFERRED_BACKENDS[0]));
}

[slaren]

You can set the devices that you want to use in llama_model_params::devices, but I don't see how that's related to Vulkan crashing.