Eval bug: GGML_ASSERT(q_to_vec_dot && "fattn: unsupported K-type") failed with Vulkan

[deiteris]

Name and Version

version: 5061 (916c83b)
built with MSVC 19.38.33134.0 for x64

Operating systems

Windows

GGML backends

Vulkan

Hardware

Ryzen 7 5800H + AMD Radeon RX 6600M 8GB

Models

Snowflake Arctic Embed L v2.0 Q8_0 (https://huggingface.co/Casual-Autopsy/snowflake-arctic-embed-l-v2.0-gguf/tree/main)
BGE Reranker v2 M3 Q8_0 (https://huggingface.co/gpustack/bge-reranker-v2-m3-GGUF/tree/main)

Problem description & steps to reproduce

When running an embedding model, llama.cpp server randomly crashes with GGML_ASSERT(q_to_vec_dot && "fattn: unsupported K-type") failed. Sometimes after first task, sometimes after several tasks.

First Bad Commit

No response

Relevant log output

.\llama-server.exe --embedding -ub 8192 -b 8192 -c 8192 --host 127.0.0.1 --port 8080 -m snowflake-arctic-embed-l-v2.0-q8_0.gguf -ngl 99 -fa -ctk q8_0 -ctv q8_0
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 6600M (AMD proprietary driver) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
build: 5061 (916c83bf) with MSVC 19.38.33134.0 for x64
system info: n_threads = 8, n_threads_batch = 8, total_threads = 16

system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |

main: binding port with default address family
main: HTTP server is listening, hostname: 127.0.0.1, port: 8080, http threads: 15
main: loading model
srv    load_model: loading model 'snowflake-arctic-embed-l-v2.0-q8_0.gguf'
llama_model_load_from_file_impl: using device Vulkan0 (AMD Radeon RX 6600M) - 8176 MiB free
llama_model_loader: loaded meta data with 36 key-value pairs and 389 tensors from snowflake-arctic-embed-l-v2.0-q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = bert
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Snowflake Arctic Embed L v2.0
llama_model_loader: - kv   3:                            general.version str              = v2.0
llama_model_loader: - kv   4:                           general.basename str              = snowflake-arctic-embed-l
llama_model_loader: - kv   5:                         general.size_label str              = 567M
llama_model_loader: - kv   6:                            general.license str              = apache-2.0
llama_model_loader: - kv   7:                               general.tags arr[str,8]       = ["sentence-transformers", "feature-ex...
llama_model_loader: - kv   8:                          general.languages arr[str,74]      = ["af", "ar", "az", "be", "bg", "bn", ...
llama_model_loader: - kv   9:                           bert.block_count u32              = 24
llama_model_loader: - kv  10:                        bert.context_length u32              = 8192
llama_model_loader: - kv  11:                      bert.embedding_length u32              = 1024
llama_model_loader: - kv  12:                   bert.feed_forward_length u32              = 4096
llama_model_loader: - kv  13:                  bert.attention.head_count u32              = 16
llama_model_loader: - kv  14:          bert.attention.layer_norm_epsilon f32              = 0.000010
llama_model_loader: - kv  15:                          general.file_type u32              = 7
llama_model_loader: - kv  16:                      bert.attention.causal bool             = false
llama_model_loader: - kv  17:                          bert.pooling_type u32              = 2
llama_model_loader: - kv  18:                       tokenizer.ggml.model str              = t5
llama_model_loader: - kv  19:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  20:                      tokenizer.ggml.tokens arr[str,250002]  = ["<s>", "<pad>", "</s>", "<unk>", ","...
llama_model_loader: - kv  21:                      tokenizer.ggml.scores arr[f32,250002]  = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  22:                  tokenizer.ggml.token_type arr[i32,250002]  = [3, 3, 3, 2, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  23:            tokenizer.ggml.add_space_prefix bool             = true
llama_model_loader: - kv  24:            tokenizer.ggml.token_type_count u32              = 1
llama_model_loader: - kv  25:    tokenizer.ggml.remove_extra_whitespaces bool             = true
llama_model_loader: - kv  26:        tokenizer.ggml.precompiled_charsmap arr[u8,237539]   = [0, 180, 2, 0, 0, 132, 0, 0, 0, 0, 0,...
llama_model_loader: - kv  27:                tokenizer.ggml.bos_token_id u32              = 0
llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  29:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  30:          tokenizer.ggml.seperator_token_id u32              = 2
llama_model_loader: - kv  31:            tokenizer.ggml.padding_token_id u32              = 1
llama_model_loader: - kv  32:               tokenizer.ggml.mask_token_id u32              = 250001
llama_model_loader: - kv  33:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  34:               tokenizer.ggml.add_eos_token bool             = true
llama_model_loader: - kv  35:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  244 tensors
llama_model_loader: - type q8_0:  145 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q8_0
print_info: file size   = 598.63 MiB (8.86 BPW)
load: model vocab missing newline token, using special_pad_id instead
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 4
load: token to piece cache size = 2.1668 MB
print_info: arch             = bert
print_info: vocab_only       = 0
print_info: n_ctx_train      = 8192
print_info: n_embd           = 1024
print_info: n_layer          = 24
print_info: n_head           = 16
print_info: n_head_kv        = 16
print_info: n_rot            = 64
print_info: n_swa            = 0
print_info: n_swa_pattern    = 1
print_info: n_embd_head_k    = 64
print_info: n_embd_head_v    = 64
print_info: n_gqa            = 1
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 1.0e-05
print_info: f_norm_rms_eps   = 0.0e+00
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 4096
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 0
print_info: pooling type     = 2
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 10000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 8192
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 0
print_info: ssm_d_inner      = 0
print_info: ssm_d_state      = 0
print_info: ssm_dt_rank      = 0
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = 335M
print_info: model params     = 566.70 M
print_info: general.name     = Snowflake Arctic Embed L v2.0
print_info: vocab type       = UGM
print_info: n_vocab          = 250002
print_info: n_merges         = 0
print_info: BOS token        = 0 '<s>'
print_info: EOS token        = 2 '</s>'
print_info: UNK token        = 3 '<unk>'
print_info: SEP token        = 2 '</s>'
print_info: PAD token        = 1 '<pad>'
print_info: MASK token       = 250001 '[PAD250000]'
print_info: LF token         = 0 '<s>'
print_info: EOG token        = 2 '</s>'
print_info: max token length = 48
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 24 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 25/25 layers to GPU
load_tensors:      Vulkan0 model buffer size =   307.22 MiB
load_tensors:   CPU_Mapped model buffer size =   291.41 MiB
......................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 8192
llama_context: n_ctx_per_seq = 8192
llama_context: n_batch       = 8192
llama_context: n_ubatch      = 8192
llama_context: causal_attn   = 0
llama_context: flash_attn    = 1
llama_context: freq_base     = 10000.0
llama_context: freq_scale    = 1
llama_context: Vulkan_Host  output buffer size =     0.00 MiB
init: kv_size = 8192, offload = 1, type_k = 'q8_0', type_v = 'q8_0', n_layer = 24, can_shift = 1
init:    Vulkan0 KV buffer size =   408.00 MiB
llama_context: KV self size  =  408.00 MiB, K (q8_0):  204.00 MiB, V (q8_0):  204.00 MiB
llama_context:    Vulkan0 compute buffer size =   512.00 MiB
llama_context: Vulkan_Host compute buffer size =   320.09 MiB
llama_context: graph nodes  = 706 (with bs=8192), 826 (with bs=1)
llama_context: graph splits = 52 (with bs=8192), 2 (with bs=1)
common_init_from_params: setting dry_penalty_last_n to ctx_size = 8192
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 8192
main: model loaded
main: chat template, chat_template: {%- for message in messages -%}
  {{- '<|im_start|>' + message.role + '
' + message.content + '<|im_end|>
' -}}
{%- endfor -%}
{%- if add_generation_prompt -%}
  {{- '<|im_start|>assistant
' -}}
{%- endif -%}, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: server is listening on http://127.0.0.1:8080 - starting the main loop
srv  update_slots: all slots are idle
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 8192, n_keep = 0, n_prompt_tokens = 11
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 11, n_tokens = 11, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 11, n_tokens = 11
slot      release: id  0 | task 0 | stop processing: n_past = 11, truncated = 0
slot launch_slot_: id  0 | task 17 | processing task
slot update_slots: id  0 | task 17 | new prompt, n_ctx_slot = 8192, n_keep = 0, n_prompt_tokens = 1024
slot update_slots: id  0 | task 17 | kv cache rm [0, end)
slot update_slots: id  0 | task 17 | prompt processing progress, n_past = 1024, n_tokens = 1024, progress = 1.000000
slot update_slots: id  0 | task 17 | prompt done, n_past = 1024, n_tokens = 1024
C:\Sources\llama.cpp\ggml\src\ggml-cpu\ggml-cpu.c:10344: GGML_ASSERT(q_to_vec_dot && "fattn: unsupported K-type") failed
C:\Sources\llama.cpp\ggml\src\ggml-cpu\ggml-cpu.c:10344:

[0cc4m]

The crash comes from the CPU backend, apparently the combination of quantized k-cache and Flash Attention is unsupported.

It fell back to the CPU cause Flash Attention on Vulkan is only supported on Nvidia GPUs with a beta driver. Without Flash Attention, only k-cache quantization is supported, so maybe give it a try without -fa and without -ctv.

[ggerganov]

Hm, that's unexpected. The CPU supports FA with quantized types - it is the reference implementation.

I am not able to reproduce this assert on my Mac after forcing -dev none, so I think something else goes wrong.

@deiteris:

• Could you check if the same commands work, starting the llama-server with -dev none
• Provide log with -lv 1 enabled when it crashes

[deiteris]

@0cc4m oh I didn't know that. I thought that latest changes to FA in Vulkan backend made it work for AMD as well. I've just noticed that CPU usage also raises when using FA, so yeah, looks like it uses CPU fallback indeed. Using just -ctk q8_0 seems to work fine so far.

@ggerganov here's the log before crash. I run the server with .\llama-server.exe --embedding -ub 4096 -b 4096 -c 4096 --host 192.168.0.2 --port 8080 -m C:\Temp\snowflake-arctic-embed-l-v2.0-q8_0.gguf -ngl 99 -fa -ctk q8_0 -ctv q8_0 -dev none -lv 1

server.log

[ggerganov]

Ok thanks, I can reproduce the bug.

[ggerganov]

@deiteris Could you test if branch #12825 works?

Btw, thanks for testing and reporting your usage of llama-server for embeddings. This workflow is not commonly exercised so this feedback is very helpful to resolve some underlying problems. Note that for example the -ctk q8_0 -ctv q8_0 arguments are now used to create a KV cache which ends up completely unused during the computation. This is something that we will fix soon and it will reduce VRAM usage significantly for this workflow.

[deiteris]

@ggerganov looking good so far! I've checked on a few large projects and don't see crashes anymore with the same parameters. Thanks!

Sure, you're welcome :) I'll see if there's anything else, but this is the only major issue I've seen for a while.