GGML_ASSERT: llama.cpp:5403: n_embd_gqa == n_embd

[Tillerino]

I am trying to start a code completion server locally with the llama-cpp-python wrapper, see here.

I am currently running this to start the server: python3 -m llama_cpp.server --model models/replit-code-v1_5-3b.Q4_0.gguf --n_ctx 16192 --n_gpu_layers -1 --use_mlock False and the program crashes with the error message

...
llm_load_tensors: ggml ctx size       =    0.07 MiB
llm_load_tensors: using CUDA for GPU acceleration
llm_load_tensors: system memory used  =   54.07 MiB
llm_load_tensors: VRAM used           = 1807.51 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
.........................................................................................
llama_new_context_with_model: n_ctx      = 16192
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_new_context_with_model: KV self size  = 2024.00 MiB, K (f16): 1012.00 MiB, V (f16): 1012.00 MiB
GGML_ASSERT: (long path)/llama.cpp:5403: n_embd_gqa == n_embd`

Now I realize I am using a downstream library to get to llama.cpp, but I figured that somebody here might be able to tell me what the assertion above even means. It might then be obvious what I'm doing wrong.

Thanks in advance

[ggerganov]

This is likely a regression from #4657 (cc @postmasters)

[Tillerino]

Ok, I'll try an older version.

[Tillerino]

Yes, 0.2.26 works just fine. Thanks for the quick answer!

[ggerganov]

When we merge #4796 you will be able to use the latest version again

[postmasters]

It is caused by my PR. So sorry about that. Thank you @ggerganov for the fix. That PR looks good to me.