Bug: Vulkan vk::DeviceLostError with multithreaded environment

[ddwkim]

What happened?

I would like to begin by expressing my sincere gratitude to the authors for their dedication and effort in developing this work.

To provide context for the issue I am encountering, I am attempting to utilize multiple threads to create multiple instances of Llama models. The objective is to improve throughput by maximizing GPU utilization. With the CUDA backend, this approach works seamlessly, achieving a 1.5 to 2.0-fold increase in throughput when using two instances compared to a single instance.

However, I am consistently facing a segmentation fault accompanied by a vk::DeviceLostError during the llama_decode function call. More specifically, the error arises when two threads execute the llama_decode function concurrently.

To aid in diagnosing this issue, I have developed a minimal example that reproduces the error. The code is available on GitHub: GitHub Code.

From my observations, the error is less likely to occur with short prompts but becomes increasingly frequent as the prompt length grows.

As I have been struggling with this issue for several days, I would greatly appreciate any insights or guidance on debugging this problem. Thank you in advance for your support and expertise.

Name and Version

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA RTX A6000 (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32
register_backend: registered backend Vulkan (1 devices)
register_device: registered device Vulkan0 (NVIDIA RTX A6000)
register_backend: registered backend CPU (1 devices)
register_device: registered device CPU (AMD Ryzen Threadripper PRO 5975WX 32-Cores)
version: 4139 (3952a22)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu

nvidia-smi: Driver Version: 535.183.06 CUDA Version: 12.2

What operating system are you seeing the problem on?

Linux

Relevant log output

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA RTX A6000 (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32
register_backend: registered backend Vulkan (1 devices)
register_device: registered device Vulkan0 (NVIDIA RTX A6000)
register_backend: registered backend CPU (1 devices)
register_device: registered device CPU (AMD Ryzen Threadripper PRO 5975WX 32-Cores)
build: 4139 (3952a221) with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu (debug)
Begin Initialize llama worker
llama_load_model_from_file: using device Vulkan0 (NVIDIA RTX A6000) - 49140 MiB free
llama_model_loader: loaded meta data with 31 key-value pairs and 147 tensors from ./Llama-3.2-1B-Instruct-GGUF/Llama-3.2-1B-Instruct-f16.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Llama 3.2 1B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Llama-3.2
llama_model_loader: - kv   5:                         general.size_label str              = 1B
llama_model_loader: - kv   6:                            general.license str              = llama3.2
llama_model_loader: - kv   7:                               general.tags arr[str,6]       = ["facebook", "meta", "pytorch", "llam...
llama_model_loader: - kv   8:                          general.languages arr[str,8]       = ["en", "de", "fr", "it", "pt", "hi", ...
llama_model_loader: - kv   9:                          llama.block_count u32              = 16
llama_model_loader: - kv  10:                       llama.context_length u32              = 131072
llama_model_loader: - kv  11:                     llama.embedding_length u32              = 2048
llama_model_loader: - kv  12:                  llama.feed_forward_length u32              = 8192
llama_model_loader: - kv  13:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv  14:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  15:                       llama.rope.freq_base f32              = 500000.000000
llama_model_loader: - kv  16:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  17:                 llama.attention.key_length u32              = 64
llama_model_loader: - kv  18:               llama.attention.value_length u32              = 64
llama_model_loader: - kv  19:                          general.file_type u32              = 1
llama_model_loader: - kv  20:                           llama.vocab_size u32              = 128256
llama_model_loader: - kv  21:                 llama.rope.dimension_count u32              = 64
llama_model_loader: - kv  22:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  23:                         tokenizer.ggml.pre str              = llama-bpe
llama_model_loader: - kv  24:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  25:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  26:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  27:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 128009
llama_model_loader: - kv  29:                    tokenizer.chat_template str              = {{- bos_token }}\n{%- if custom_tools ...
llama_model_loader: - kv  30:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   34 tensors
llama_model_loader: - type  f16:  113 tensors
llm_load_vocab: special tokens cache size = 256
llm_load_vocab: token to piece cache size = 0.7999 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 128256
llm_load_print_meta: n_merges         = 280147
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 131072
llm_load_print_meta: n_embd           = 2048
llm_load_print_meta: n_layer          = 16
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 64
llm_load_print_meta: n_embd_head_v    = 64
llm_load_print_meta: n_gqa            = 4
llm_load_print_meta: n_embd_k_gqa     = 512
llm_load_print_meta: n_embd_v_gqa     = 512
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 8192
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 500000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 131072
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = 1B
llm_load_print_meta: model ftype      = F16
llm_load_print_meta: model params     = 1.24 B
llm_load_print_meta: model size       = 2.30 GiB (16.00 BPW)
llm_load_print_meta: general.name     = Llama 3.2 1B Instruct
llm_load_print_meta: BOS token        = 128000 '<|begin_of_text|>'
llm_load_print_meta: EOS token        = 128009 '<|eot_id|>'
llm_load_print_meta: EOT token        = 128009 '<|eot_id|>'
llm_load_print_meta: EOM token        = 128008 '<|eom_id|>'
llm_load_print_meta: LF token         = 128 'Ä'
llm_load_print_meta: EOG token        = 128008 '<|eom_id|>'
llm_load_print_meta: EOG token        = 128009 '<|eot_id|>'
llm_load_print_meta: max token length = 256
ggml_vulkan: Compiling shaders..............................Done!
llm_load_tensors: offloading 16 repeating layers to GPU
llm_load_tensors: offloading output layer to GPU
llm_load_tensors: offloaded 17/17 layers to GPU
llm_load_tensors:      Vulkan0 model buffer size =  2357.26 MiB
llm_load_tensors:   CPU_Mapped model buffer size =   501.00 MiB
............................................................
llama_new_context_with_model: n_seq_max     = 1
llama_new_context_with_model: n_ctx         = 4096
llama_new_context_with_model: n_ctx_per_seq = 4096
llama_new_context_with_model: n_batch       = 2500
llama_new_context_with_model: n_ubatch      = 512
llama_new_context_with_model: flash_attn    = 0
llama_new_context_with_model: freq_base     = 500000.0
llama_new_context_with_model: freq_scale    = 1
llama_new_context_with_model: n_ctx_per_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_kv_cache_init:    Vulkan0 KV buffer size =   128.00 MiB
llama_new_context_with_model: KV self size  =  128.00 MiB, K (f16):   64.00 MiB, V (f16):   64.00 MiB
llama_new_context_with_model: Vulkan_Host  output buffer size =     0.49 MiB
llama_new_context_with_model:    Vulkan0 compute buffer size =   280.00 MiB
llama_new_context_with_model: Vulkan_Host compute buffer size =    12.01 MiB
llama_new_context_with_model: graph nodes  = 518
llama_new_context_with_model: graph splits = 2
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
Initialized llama worker
Begin Initialize llama worker
llama_load_model_from_file: using device Vulkan0 (NVIDIA RTX A6000) - 49140 MiB free
llama_model_loader: loaded meta data with 31 key-value pairs and 147 tensors from ./Llama-3.2-1B-Instruct-GGUF/Llama-3.2-1B-Instruct-f16.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Llama 3.2 1B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Llama-3.2
llama_model_loader: - kv   5:                         general.size_label str              = 1B
llama_model_loader: - kv   6:                            general.license str              = llama3.2
llama_model_loader: - kv   7:                               general.tags arr[str,6]       = ["facebook", "meta", "pytorch", "llam...
llama_model_loader: - kv   8:                          general.languages arr[str,8]       = ["en", "de", "fr", "it", "pt", "hi", ...
llama_model_loader: - kv   9:                          llama.block_count u32              = 16
llama_model_loader: - kv  10:                       llama.context_length u32              = 131072
llama_model_loader: - kv  11:                     llama.embedding_length u32              = 2048
llama_model_loader: - kv  12:                  llama.feed_forward_length u32              = 8192
llama_model_loader: - kv  13:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv  14:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  15:                       llama.rope.freq_base f32              = 500000.000000
llama_model_loader: - kv  16:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  17:                 llama.attention.key_length u32              = 64
llama_model_loader: - kv  18:               llama.attention.value_length u32              = 64
llama_model_loader: - kv  19:                          general.file_type u32              = 1
llama_model_loader: - kv  20:                           llama.vocab_size u32              = 128256
llama_model_loader: - kv  21:                 llama.rope.dimension_count u32              = 64
llama_model_loader: - kv  22:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  23:                         tokenizer.ggml.pre str              = llama-bpe
llama_model_loader: - kv  24:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  25:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  26:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  27:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 128009
llama_model_loader: - kv  29:                    tokenizer.chat_template str              = {{- bos_token }}\n{%- if custom_tools ...
llama_model_loader: - kv  30:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   34 tensors
llama_model_loader: - type  f16:  113 tensors
llm_load_vocab: special tokens cache size = 256
llm_load_vocab: token to piece cache size = 0.7999 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 128256
llm_load_print_meta: n_merges         = 280147
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 131072
llm_load_print_meta: n_embd           = 2048
llm_load_print_meta: n_layer          = 16
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 64
llm_load_print_meta: n_embd_head_v    = 64
llm_load_print_meta: n_gqa            = 4
llm_load_print_meta: n_embd_k_gqa     = 512
llm_load_print_meta: n_embd_v_gqa     = 512
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 8192
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 500000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 131072
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = 1B
llm_load_print_meta: model ftype      = F16
llm_load_print_meta: model params     = 1.24 B
llm_load_print_meta: model size       = 2.30 GiB (16.00 BPW)
llm_load_print_meta: general.name     = Llama 3.2 1B Instruct
llm_load_print_meta: BOS token        = 128000 '<|begin_of_text|>'
llm_load_print_meta: EOS token        = 128009 '<|eot_id|>'
llm_load_print_meta: EOT token        = 128009 '<|eot_id|>'
llm_load_print_meta: EOM token        = 128008 '<|eom_id|>'
llm_load_print_meta: LF token         = 128 'Ä'
llm_load_print_meta: EOG token        = 128008 '<|eom_id|>'
llm_load_print_meta: EOG token        = 128009 '<|eot_id|>'
llm_load_print_meta: max token length = 256
llm_load_tensors: offloading 16 repeating layers to GPU
llm_load_tensors: offloading output layer to GPU
llm_load_tensors: offloaded 17/17 layers to GPU
llm_load_tensors:      Vulkan0 model buffer size =  2357.26 MiB
llm_load_tensors:   CPU_Mapped model buffer size =   501.00 MiB
............................................................
llama_new_context_with_model: n_seq_max     = 1
llama_new_context_with_model: n_ctx         = 4096
llama_new_context_with_model: n_ctx_per_seq = 4096
llama_new_context_with_model: n_batch       = 2500
llama_new_context_with_model: n_ubatch      = 512
llama_new_context_with_model: flash_attn    = 0
llama_new_context_with_model: freq_base     = 500000.0
llama_new_context_with_model: freq_scale    = 1
llama_new_context_with_model: n_ctx_per_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_kv_cache_init:    Vulkan0 KV buffer size =   128.00 MiB
llama_new_context_with_model: KV self size  =  128.00 MiB, K (f16):   64.00 MiB, V (f16):   64.00 MiB
llama_new_context_with_model: Vulkan_Host  output buffer size =     0.49 MiB
llama_new_context_with_model:    Vulkan0 compute buffer size =   280.00 MiB
llama_new_context_with_model: Vulkan_Host compute buffer size =    12.01 MiB
llama_new_context_with_model: graph nodes  = 518
llama_new_context_with_model: graph splits = 2
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
Initialized llama worker
Workers initialized
Inference of job 0 by worker  1
Jobs pushed to queue
Inference of job 1 by worker  0
Prompt tokens size: 282
Parsed input in 2 ms
Prompt tokens size: 282
Parsed input in 2 ms
Decoded in 3885 ms
Worker 0 processed job successfully.
Inference of job 2 by worker  0
Prompt tokens size: 282
Parsed input in 4 ms
Decoded in 4360 ms
Worker 1 processed job successfully.
Inference of job 3 by worker  1
Prompt tokens size: 282
Parsed input in 4 ms
Decoded in 3988 ms
Worker 1 processed job successfully.
Inference of job 4 by worker  1
Prompt tokens size: 282
Parsed input in 3 ms
Decoded in 4773 ms
Worker 0 processed job successfully.
Inference of job 5 by worker  0
Prompt tokens size: 282
Parsed input in 4 ms
Decoded in 2668 ms
Worker 0 processed job successfully.
Inference of job 6 by worker  0
Prompt tokens size: 282
Parsed input in 4 ms
Decoded in 4267 ms
Worker 1 processed job successfully.
Inference of job 7 by worker  1
Prompt tokens size: 282
Parsed input in 2 ms
[1]    659873 segmentation fault (core dumped)  ./build/bin/llama_simple_worker -m  -n 256 -b 2500 -ub 512 --temp 0.5 -t 2 -t

[0cc4m]

Thank you for trying this and the example. There's been a previous issue about this that I haven't gotten around to yet, but I can't find it right now.

Basically someone has to go through the Vulkan code to make sure all global variables and functions are thread-safe and can handle multiple accesses at the same time. This is not trivial, so I haven't gotten around to it yet.

[ddwkim]

I have installed and run with validation layer, and the log is as follows. I hope this helps.

ggml_pipeline_cleanup(0x7f96ac1746f0)
silu_f32)
ggml_pipeline_cleanup(ggml_vk_ctx_begin(Vulkan0)
ggml_vk_create_cmd_buffer()
contig_cpy_f32_f32)
ggml_pipeline_cleanup(mul_mat_vec_q4_k_f32_f32)
ggml_pipeline_cleanup(mul_mat_vec_nc_f16_f32)
ggml_pipeline_cleanup(soft_max_f32)
ggml_pipeline_cleanup(add_f32_norepeat)
ggml_pipeline_cleanup(contig_cpy_f32_f16)
ggml_pipeline_cleanup(mul_mat_vec_q6_k_f32_f32)
ggml_pipeline_cleanup(ggml_vk_op_f32((0x62a32cdaa200, name=norm-30, type=0, ne0=0x3c0, ne1=0x1, ne2=0x1, ne3=0x1, nb0=0x4, nb1=0xf00, nb2=0xf00, nb3=0xf00), (0x62a32ccf8790, name=blk.30.ffn_norm.weight, type=0, ne0=0x3c0, ne1=0x1, ne2=0x1, ne3=0x1, nb0=mul_mat_vec_q8_0_f32_f32)
ggml_pipeline_cleanup(rope_norm_f32)
ggml_pipeline_cleanup(mul_mat_vec_q5_0_f32_f32)
0x4, nb1=0xf00, nb2=ggml_pipeline_cleanup(cpy_f32_f16)
ggml_pipeline_cleanup(mul_f32_norepeat)
ggml_pipeline_cleanup(mul_mat_vec_p021_f16_f32)
ggml_pipeline_cleanup(rms_norm_f32)
0xf00, nb3=0xf00), (0x62a32cdaa370, name=ffn_norm-30, type=0, ne0=0x3c0, ne1=0x1, ne2=0x1, ne3=0x1ggml_vk_queue_cleanup()
, nb0=0x4, nb1=0xf00, nb2=0xf00, nb3=0xf00), MUL, )
ggml_vk_sync_buffers()
VUID-vkResetCommandPool-commandPool-00040(ERROR / SPEC): msgNum: -1254218959 - Validation Error: [ VUID-vkResetCommandPool-commandPool-00040 ] Object 0: handle = 0x7f97c425c440, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0xfd5b260000000001, type = VK_OBJECT_TYPE_COMMAND_POOL; | MessageID = 0xb53e2331 | vkResetCommandPool():  (VkCommandBuffer 0x7f97c425c440[]) is in use.
The Vulkan spec states: All VkCommandBuffer objects allocated from commandPool must not be in the pending state (https://vulkan.lunarg.com/doc/view/1.3.296.0/linux/1.3-extensions/vkspec.html#VUID-vkResetCommandPool-commandPool-00040)
    Objects: 2
        [0] 0x7f97c425c440, type: 6, name: NULL
        [1] 0xfd5b260000000001, type: 25, name: NULL
ggml_vk_dispatch_pipeline(mul_f32_norepeat, {(0x637a9d0000000a52, 0, 0xf00), (0x763bcc0000000a4b, 0xf454200, 0xf00), (0x637a9d0000000a52, 0, 0xf00), }, (0x1,0x2,0x1))
[1]    1083743 segmentation fault (core dumped)

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.