Eval bug: llama.cpp CPU bound while inferencing against DeepSeek-R1 GGUF

[chrisjob1021]

Name and Version

$ ./build/bin/llama-cli --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA L40S, compute capability 8.9, VMM: yes
version: 4625 (5598f47)
built with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu

Operating systems

Linux

GGML backends

CUDA

Hardware

Intel(R) Xeon(R) w5-3425 + NVIDIA L40S

Models

unsloth/DeepSeek-R1-GGUF

Problem description & steps to reproduce

When attempting to use llama-cli to inference, it becomes CPU bound and is painfully slow (less than one token per second). nvtop shows that the GPU is 0% utilized (all CPU being used) despite 14 layers and 44GB offloaded to VRAM. I'm following the instructions outlined on Unsloth's blog and running the following command:
!build/bin/llama-cli \ --model DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf \ --cache-type-k q4_0 \ --threads 64 \ --prio 2 \ --temp 0.6 \ --ctx-size 8192 \ --seed 3407 \ --n-gpu-layers 16 \ -no-cnv \ --prompt "<｜User｜>Create a Flappy Bird game in Python.<｜Assistant｜>"

First Bad Commit

No response

Relevant log output

llama_model_load_from_file_impl: using device CUDA0 (NVIDIA L40S) - 45055 MiB free
llama_model_loader: additional 2 GGUFs metadata loaded.
llama_model_loader: loaded meta data with 52 key-value pairs and 1025 tensors from DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = DeepSeek R1 BF16
llama_model_loader: - kv   3:                       general.quantized_by str              = Unsloth
llama_model_loader: - kv   4:                         general.size_label str              = 256x20B
llama_model_loader: - kv   5:                           general.repo_url str              = https://huggingface.co/unsloth
llama_model_loader: - kv   6:                      deepseek2.block_count u32              = 61
llama_model_loader: - kv   7:                   deepseek2.context_length u32              = 163840
llama_model_loader: - kv   8:                 deepseek2.embedding_length u32              = 7168
llama_model_loader: - kv   9:              deepseek2.feed_forward_length u32              = 18432
llama_model_loader: - kv  10:             deepseek2.attention.head_count u32              = 128
llama_model_loader: - kv  11:          deepseek2.attention.head_count_kv u32              = 128
llama_model_loader: - kv  12:                   deepseek2.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv  13: deepseek2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  14:                deepseek2.expert_used_count u32              = 8
llama_model_loader: - kv  15:        deepseek2.leading_dense_block_count u32              = 3
llama_model_loader: - kv  16:                       deepseek2.vocab_size u32              = 129280
llama_model_loader: - kv  17:            deepseek2.attention.q_lora_rank u32              = 1536
llama_model_loader: - kv  18:           deepseek2.attention.kv_lora_rank u32              = 512
llama_model_loader: - kv  19:             deepseek2.attention.key_length u32              = 192
llama_model_loader: - kv  20:           deepseek2.attention.value_length u32              = 128
llama_model_loader: - kv  21:       deepseek2.expert_feed_forward_length u32              = 2048
llama_model_loader: - kv  22:                     deepseek2.expert_count u32              = 256
llama_model_loader: - kv  23:              deepseek2.expert_shared_count u32              = 1
llama_model_loader: - kv  24:             deepseek2.expert_weights_scale f32              = 2.500000
llama_model_loader: - kv  25:              deepseek2.expert_weights_norm bool             = true
llama_model_loader: - kv  26:               deepseek2.expert_gating_func u32              = 2
llama_model_loader: - kv  27:             deepseek2.rope.dimension_count u32              = 64
llama_model_loader: - kv  28:                deepseek2.rope.scaling.type str              = yarn
llama_model_loader: - kv  29:              deepseek2.rope.scaling.factor f32              = 40.000000
llama_model_loader: - kv  30: deepseek2.rope.scaling.original_context_length u32              = 4096
llama_model_loader: - kv  31: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.100000
llama_model_loader: - kv  32:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  33:                         tokenizer.ggml.pre str              = deepseek-v3
llama_model_loader: - kv  34:                      tokenizer.ggml.tokens arr[str,129280]  = ["<｜begin▁of▁sentence｜>", "<�...
llama_model_loader: - kv  35:                  tokenizer.ggml.token_type arr[i32,129280]  = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  36:                      tokenizer.ggml.merges arr[str,127741]  = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e...
llama_model_loader: - kv  37:                tokenizer.ggml.bos_token_id u32              = 0
llama_model_loader: - kv  38:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  39:            tokenizer.ggml.padding_token_id u32              = 128815
llama_model_loader: - kv  40:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  41:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  42:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
llama_model_loader: - kv  43:               general.quantization_version u32              = 2
llama_model_loader: - kv  44:                          general.file_type u32              = 24
llama_model_loader: - kv  45:                      quantize.imatrix.file str              = DeepSeek-R1.imatrix
llama_model_loader: - kv  46:                   quantize.imatrix.dataset str              = /training_data/calibration_datav3.txt
llama_model_loader: - kv  47:             quantize.imatrix.entries_count i32              = 720
llama_model_loader: - kv  48:              quantize.imatrix.chunks_count i32              = 124
llama_model_loader: - kv  49:                                   split.no u16              = 0
llama_model_loader: - kv  50:                        split.tensors.count i32              = 1025
llama_model_loader: - kv  51:                                split.count u16              = 3
llama_model_loader: - type  f32:  361 tensors
llama_model_loader: - type q4_K:  190 tensors
llama_model_loader: - type q5_K:  116 tensors
llama_model_loader: - type q6_K:  184 tensors
llama_model_loader: - type iq2_xxs:    6 tensors
llama_model_loader: - type iq1_s:  168 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = IQ1_S - 1.5625 bpw
print_info: file size   = 130.60 GiB (1.67 BPW) 
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 819
load: token to piece cache size = 0.8223 MB
print_info: arch             = deepseek2
print_info: vocab_only       = 0
print_info: n_ctx_train      = 163840
print_info: n_embd           = 7168
print_info: n_layer          = 61
print_info: n_head           = 128
print_info: n_head_kv        = 128
print_info: n_rot            = 64
print_info: n_swa            = 0
print_info: n_embd_head_k    = 192
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 1
print_info: n_embd_k_gqa     = 24576
print_info: n_embd_v_gqa     = 16384
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: n_ff             = 18432
print_info: n_expert         = 256
print_info: n_expert_used    = 8
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = yarn
print_info: freq_base_train  = 10000.0
print_info: freq_scale_train = 0.025
print_info: n_ctx_orig_yarn  = 4096
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 0
print_info: ssm_d_inner      = 0
print_info: ssm_d_state      = 0
print_info: ssm_dt_rank      = 0
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = 671B
print_info: model params     = 671.03 B
print_info: general.name     = DeepSeek R1 BF16
print_info: n_layer_dense_lead   = 3
print_info: n_lora_q             = 1536
print_info: n_lora_kv            = 512
print_info: n_ff_exp             = 2048
print_info: n_expert_shared      = 1
print_info: expert_weights_scale = 2.5
print_info: expert_weights_norm  = 1
print_info: expert_gating_func   = sigmoid
print_info: rope_yarn_log_mul    = 0.1000
print_info: vocab type       = BPE
print_info: n_vocab          = 129280
print_info: n_merges         = 127741
print_info: BOS token        = 0 '<｜begin▁of▁sentence｜>'
print_info: EOS token        = 1 '<｜end▁of▁sentence｜>'
print_info: EOT token        = 1 '<｜end▁of▁sentence｜>'
print_info: PAD token        = 128815 '<｜PAD▁TOKEN｜>'
print_info: LF token         = 201 'Ċ'
print_info: FIM PRE token    = 128801 '<｜fim▁begin｜>'
print_info: FIM SUF token    = 128800 '<｜fim▁hole｜>'
print_info: FIM MID token    = 128802 '<｜fim▁end｜>'
print_info: EOG token        = 1 '<｜end▁of▁sentence｜>'
print_info: max token length = 256
load_tensors: offloading 16 repeating layers to GPU
load_tensors: offloaded 16/62 layers to GPU
load_tensors:        CUDA0 model buffer size = 35892.95 MiB
load_tensors:          AMX model buffer size =  7640.76 MiB
load_tensors:   CPU_Mapped model buffer size = 46321.61 MiB
load_tensors:   CPU_Mapped model buffer size = 47098.01 MiB
load_tensors:   CPU_Mapped model buffer size =  3659.96 MiB
llama_init_from_model: n_seq_max     = 1
llama_init_from_model: n_ctx         = 8192
llama_init_from_model: n_ctx_per_seq = 8192
llama_init_from_model: n_batch       = 2048
llama_init_from_model: n_ubatch      = 512
llama_init_from_model: flash_attn    = 0
llama_init_from_model: freq_base     = 10000.0
llama_init_from_model: freq_scale    = 0.025
llama_init_from_model: n_ctx_per_seq (8192) < n_ctx_train (163840) -- the full capacity of the model will not be utilized
llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'q4_0', type_v = 'f16', n_layer = 61, can_shift = 0
llama_kv_cache_init:      CUDA0 KV buffer size =  5824.00 MiB
llama_kv_cache_init:        CPU KV buffer size = 16380.00 MiB
llama_init_from_model: KV self size  = 22204.00 MiB, K (q4_0): 6588.00 MiB, V (f16): 15616.00 MiB
llama_init_from_model:        CPU  output buffer size =     0.49 MiB
llama_init_from_model:      CUDA0 compute buffer size =  2218.00 MiB
llama_init_from_model:  CUDA_Host compute buffer size =  2193.01 MiB
llama_init_from_model: graph nodes  = 5025
llama_init_from_model: graph splits = 754 (with bs=512), 3 (with bs=1)
common_init_from_params: KV cache shifting is not supported for this model, disabling KV cache shifting
common_init_from_params: setting dry_penalty_last_n to ctx_size = 8192
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
main: llama threadpool init, n_threads = 64

system_info: n_threads = 64 (n_threads_batch = 64) / 24 | CUDA : ARCHS = 890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | AMX_INT8 = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 | 

sampler seed: 3407
sampler params: 
	repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
	dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 8192
	top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, temp = 0.600
	mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampler chain: logits -> logit-bias -> penalties -> dry -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist 
generate: n_ctx = 8192, n_batch = 2048, n_predict = -1, n_keep = 1

Create a Flappy Bird game in Python.<think>
Okay, the user wants me to create a Flappy Bird game in Python. Let me think about how to approach this.

[cpumaxx]

What do top and nvidia-smi tell you about vram usage? If its in use, the gpu is almost certainly being used for whatever part of the model is loaded into vram.
As a sanity test: What is the performance with -ngl 8 or -ngl 0?
Do you see a decrease in performance as you run with fewer layers, or does it stay constant?

[chrisjob1021]

You were right with the the sanity test. I'm seeing a decrease in performance, but it's not appreciably faster with -ngl 8 or 16.

Here's nvtop during inferencing:
Device 0 [NVIDIA L40S] PCIe GEN 4@16x RX: 0.000 KiB/s TX: 1.000 KiB/s
GPU 2520MHz MEM 9000MHz TEMP 58°C FAN N/A% POW 96 / 350 W
GPU[ 0%] MEM[||||||||||||43.917Gi/44.988Gi]

PID        USER DEV     TYPE  GPU        GPU MEM    CPU  HOST MEM Command                                                   

29457 0 Compute 0% 44378MiB 96% 1119% 52072MiB build/bin/llama-cli --model DeepSeek-R1-GGUF/DeepSeek-R1-U

And here's nvidia-smi:
Tue Feb 4 23:38:11 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03 Driver Version: 560.35.03 CUDA Version: 12.6 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA L40S Off | 00000000:34:00.0 Off | 0 |
| N/A 59C P0 105W / 350W | 44439MiB / 46068MiB | 8% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 29457 C build/bin/llama-cli 44426MiB |
+-----------------------------------------------------------------------------------------+

With -ngl 0 in 90 seconds, this is how far I get:
Create a Flappy Bird game in Python.
Okay, the user wants me to create a Fl

With -ngl 8 in 90 seconds:
Create a Flappy Bird game in Python.
Okay, the user wants me to create a Flappy Bird game in Python. Let me think
llama_perf_sampler_print: sampling time = 2.35 ms / 33 runs ( 0.07 ms per token, 14018.69 tokens per second)
llama_perf_context_print: load time = 54651.57 ms
llama_perf_context_print: prompt eval time = 9566.60 ms / 12 tokens ( 797.22 ms per token, 1.25 tokens per second)
llama_perf_context_print: eval time = 24334.76 ms / 20 runs ( 1216.74 ms per token, 0.82 tokens per second)
llama_perf_context_print: total time = 34860.87 ms / 32 tokens
Interrupted by user

With -ngl 16 in 90 seconds:
Create a Flappy Bird game in Python.
Okay, the user wants me to create a Flappy Bird game in Python. Let me think about how to approach this.
llama_perf_sampler_print: sampling time = 2.75 ms / 39 runs ( 0.07 ms per token, 14192.14 tokens per second)
llama_perf_context_print: load time = 56269.57 ms
llama_perf_context_print: prompt eval time = 6522.99 ms / 12 tokens ( 543.58 ms per token, 1.84 tokens per second)
llama_perf_context_print: eval time = 26513.44 ms / 26 runs ( 1019.75 ms per token, 0.98 tokens per second)
llama_perf_context_print: total time = 33258.56 ms / 38 tokens
Interrupted by user

[lingster]

have you tried reducing the --threads from 64 to say 16? I have a 64 core amd cpu and I found optimal performance around this point.

if you have enough system memory and ok with a longer startup time you can add --no-mmap which will load the model into system ram, rather than have it mapped to disk.

[chrisjob1021]

Thanks for the suggestions. I have tried both options and neither were appreciably faster.

[lingster]

If you have access to other gpus on another server you could try using the rpc mode. And get more layers loaded into gpu vram.

Also try to reduce your context size to either 4k or 2k

[ubergarm]

@chrisjob1021

The Intel Xeon w5-3425 supposedly has AMX extensions...

A couple days ago ktransformers dropped a python binary wheel that won't run on my AMD rigs:

Intel AMX Optimization – Our AMX-accelerated kernel is meticulously tuned, running several times faster than existing llama.cpp implementations. We plan to open-source this kernel after cleansing and are considering upstream contributions to llama.cpp.

I came here looking to see if they had opened a PR with the cleansed kernel yet. I have a rough guide on how to try it here.

[lingster]

@ubergarm : thanks for sharing your guide. I'm trying to offload the suggested layers from the ktransformers configs via the new --override-tensors options. You maybe interested to join in the discussion here: #11397 (comment). I'll be posting my test results there.

[chrisjob1021]

@lingster thanks I'll give that a try. @ubergarm I'll also check that out and can report back

[ubergarm]

@lingster @chrisjob1021 I opened a Q&A including benchmarks for a big o' dual socket Intel Xeon 6980P that I wanna get running faster. #12088

Cheers!

[lingster]

Love this detailed post. Can you try running with a lower thread count? On my 64 core rig, 16 threads seemed to be a good sweetspot.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.