Misc. bug: [server] Using q8_0 for KV cache reduces performance when also using a draft model

[mostlygeek]

Name and Version

$ /mnt/nvme/llama-server/llama-server-be0e35 --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 4 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  Device 1: Tesla P40, compute capability 6.1, VMM: yes
  Device 2: Tesla P40, compute capability 6.1, VMM: yes
  Device 3: Tesla P40, compute capability 6.1, VMM: yes
version: 4187 (be0e350c)
built with cc (Ubuntu 13.2.0-23ubuntu4) 13.2.0 for x86_64-linux-gnu

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Problem description & steps to reproduce

There is a pretty consistent 16% tokens/second performance drop when using --cache-type-k q8_0 --cache-type-v q8_0 with a draft model. It doesn't happen if I don't use a draft model.

model	python	typescript	swift
qwen-coder-32b-q4	79.9	54.48	46.67
qwen-coder-32b-q4-kv	66.60 (-16.6%)	45.27 (-16%)	39.24 (-15.9%)

The test I am using is to prompt for a snake game to be written in python, typescript and swift.

for model in "qwen-coder-32b-q4" "qwen-coder-32b-q4-kv"; do 
    for lang in "python" "typescript" "swift"; do
        echo "Generating Snake Game in $lang using $model"
        curl -s --url http://localhost:8080/v1/chat/completions -d "{\"messages\": [{\"role\": \"system\", \"content\": \"you only write code.\"}, {\"role\": \"user\", \"content\": \"write snake game in $lang\"}], \"temperature\": 0.1, \"model\":\"$model\"}" > /dev/null
    done
done

Here is my llama-swap configuration for the models above. The only difference is the addition of the cache-type flags:

models:
  "qwen-coder-32b-q4":
    # main model on 3090, draft on P40 #1
    #
    # gist results: python: 79.97 tps, typescript: 54.48 tps, swift: 46.67 tps
    cmd: >
      /mnt/nvme/llama-server/llama-server-be0e35
      --host 127.0.0.1 --port 9503
      --flash-attn --metrics
      --slots
      --model /mnt/nvme/models/Qwen2.5-Coder-32B-Instruct-Q4_K_M.gguf
      -ngl 99
      --ctx-size 19000
      --model-draft /mnt/nvme/models/Qwen2.5-Coder-0.5B-Instruct-Q8_0.gguf
      -ngld 99
      --draft-max 16
      --draft-min 4
      --draft-p-min 0.4
      --device CUDA0
      --device-draft CUDA1
    proxy: "http://127.0.0.1:9503"

  "qwen-coder-32b-q4-kv":
    # main model on 3090, draft on P40 #1
    #
    # gist results: python: 66.60, typescript 45.27, swift 39.24
    cmd: >
      /mnt/nvme/llama-server/llama-server-be0e35
      --host 127.0.0.1 --port 9503
      --flash-attn --metrics
      --slots
      --model /mnt/nvme/models/Qwen2.5-Coder-32B-Instruct-Q4_K_M.gguf
      -ngl 99
      --ctx-size 19000
      --model-draft /mnt/nvme/models/Qwen2.5-Coder-0.5B-Instruct-Q8_0.gguf
      -ngld 99
      --draft-max 16
      --draft-min 4
      --draft-p-min 0.4
      --device CUDA0
      --device-draft CUDA1
      --cache-type-k q8_0 --cache-type-v q8_0        <-- ADDED THESE FLAGS
    proxy: "http://127.0.0.1:9503"

Raw test data:

# qwen-coder-32b-q4 (python, typescript, swift)
prompt eval time =      64.50 ms /     6 tokens (   10.75 ms per token,    93.02 tokens per second)
       eval time =   11264.22 ms /   900 tokens (   12.52 ms per token,    79.90 tokens per second)       
prompt eval time =      68.14 ms /     7 tokens (    9.73 ms per token,   102.73 tokens per second)
       eval time =   15766.30 ms /   859 tokens (   18.35 ms per token,    54.48 tokens per second)       
prompt eval time =      61.53 ms /     6 tokens (   10.25 ms per token,    97.51 tokens per second)
       eval time =   19349.37 ms /   903 tokens (   21.43 ms per token,    46.67 tokens per second)


# qwen-coder-32b-q4-kv (python, typescript, swift)
prompt eval time =      52.95 ms /    23 tokens (    2.30 ms per token,   434.37 tokens per second)
       eval time =   13513.06 ms /   900 tokens (   15.01 ms per token,    66.60 tokens per second)       
prompt eval time =      69.98 ms /     7 tokens (   10.00 ms per token,   100.03 tokens per second)
       eval time =   19462.99 ms /   881 tokens (   22.09 ms per token,    45.27 tokens per second)       
prompt eval time =      63.20 ms /     6 tokens (   10.53 ms per token,    94.94 tokens per second)
       eval time =   27041.86 ms /  1061 tokens (   25.49 ms per token,    39.24 tokens per second)

First Bad Commit

No response

Relevant log output

No response

[ggerganov]

Likely the draft model should keep using F16 KV cache, otherwise it might be hitting some sub-optimal FA kernels. The change is pretty trivial in server.cpp - you can give it a try.

[mostlygeek]

I haven’t looked too deeply at the code before. Would you suggest that the draft model and the main model have separate settings for the kv cache type, or that the draft model always use F16?

[ggerganov]

The draft model can be configured to use F16 KV cache regardless of the type selected for the target model. This is not exposed through the current CLI settings, so here is a patch to see if it makes a difference: #10586

[mostlygeek]

Thanks for the quick PR.

Forcing the draft models cache did not make a difference. I ran my tests again with the latest commit on master and compared it to the branch:

without -cache-type-[k/v], version: 4230 (0c39f44d):

prompt eval time =      52.00 ms /    23 tokens (    2.26 ms per token,   442.32 tokens per second)
       eval time =   10921.28 ms /   900 tokens (   12.13 ms per token,    82.41 tokens per second)
prompt eval time =      66.86 ms /     7 tokens (    9.55 ms per token,   104.69 tokens per second)
       eval time =   15883.79 ms /   931 tokens (   17.06 ms per token,    58.61 tokens per second)
prompt eval time =      60.51 ms /     6 tokens (   10.08 ms per token,    99.16 tokens per second)
       eval time =   21301.38 ms /  1014 tokens (   21.01 ms per token,    47.60 tokens per second)

with -cache-type-[k/v], version: 4230 (0c39f44d):

prompt eval time =      52.75 ms /    23 tokens (    2.29 ms per token,   436.03 tokens per second)
       eval time =   13575.56 ms /   900 tokens (   15.08 ms per token,    66.30 tokens per second)
prompt eval time =      68.25 ms /     7 tokens (    9.75 ms per token,   102.56 tokens per second)
       eval time =   19342.16 ms /   811 tokens (   23.85 ms per token,    41.93 tokens per second)
prompt eval time =      61.41 ms /     6 tokens (   10.24 ms per token,    97.70 tokens per second)
       eval time =   28555.61 ms /  1066 tokens (   26.79 ms per token,    37.33 tokens per second)

force KV F16 with #10586 PR, version 4225 (150d6e92):

prompt eval time =      52.89 ms /    23 tokens (    2.30 ms per token,   434.89 tokens per second)
       eval time =   13434.22 ms /   900 tokens (   14.93 ms per token,    66.99 tokens per second)
prompt eval time =      69.01 ms /     7 tokens (    9.86 ms per token,   101.44 tokens per second)
       eval time =   19370.07 ms /   895 tokens (   21.64 ms per token,    46.21 tokens per second)
prompt eval time =      63.17 ms /     6 tokens (   10.53 ms per token,    94.98 tokens per second)
       eval time =   25910.73 ms /   959 tokens (   27.02 ms per token,    37.01 tokens per second)

[ggerganov]

Post the results from the following commands:

llama-bench --model /mnt/nvme/models/Qwen2.5-Coder-32B-Instruct-Q4_K_M.gguf -p 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16 -n 0 -ngl 99 -fa 1

llama-bench --model /mnt/nvme/models/Qwen2.5-Coder-32B-Instruct-Q4_K_M.gguf -p 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16 -n 0 -ngl 99 -fa 1 -ctk q8_0 -ctv q8_0

[mostlygeek]

(all GPUS) 3xP40, 1x3090

$ ./llama-bench --model /mnt/nvme/models/Qwen2.5-Coder-32B-Instruct-Q4_K_M.gguf -p 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16 -n 0 -ngl 99 -fa 1
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 4 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  Device 1: Tesla P40, compute capability 6.1, VMM: yes
  Device 2: Tesla P40, compute capability 6.1, VMM: yes
  Device 3: Tesla P40, compute capability 6.1, VMM: yes
| model                          |       size |     params | backend    | ngl | fa |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | ------------: | -------------------: |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp1 |         13.69 ± 0.09 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp2 |         25.56 ± 0.06 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp3 |         32.03 ± 0.12 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp4 |         34.49 ± 0.16 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp5 |         37.22 ± 0.17 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp6 |         39.03 ± 0.04 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp7 |         40.30 ± 0.05 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp8 |         39.72 ± 0.03 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp9 |         74.43 ± 1.00 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp10 |         82.42 ± 0.51 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp11 |         93.91 ± 0.56 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp12 |        101.44 ± 0.29 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp13 |        110.27 ± 0.53 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp14 |        117.69 ± 0.58 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp15 |        125.66 ± 0.16 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp16 |        135.71 ± 0.57 |

build: 0c39f44d (4230)

$ ./llama-bench --model /mnt/nvme/models/Qwen2.5-Coder-32B-Instruct-Q4_K_M.gguf -p 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16 -n 0 -ngl 99 -fa 1 -ctk q8_0 -ctv q8_0
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 4 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  Device 1: Tesla P40, compute capability 6.1, VMM: yes
  Device 2: Tesla P40, compute capability 6.1, VMM: yes
  Device 3: Tesla P40, compute capability 6.1, VMM: yes
| model                          |       size |     params | backend    | ngl | type_k | type_v | fa |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | -: | ------------: | -------------------: |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |           pp1 |         13.42 ± 0.08 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |           pp2 |         25.27 ± 0.01 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |           pp3 |         31.56 ± 0.50 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |           pp4 |         33.27 ± 0.05 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |           pp5 |         36.50 ± 0.15 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |           pp6 |         38.26 ± 0.16 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |           pp7 |         39.11 ± 0.06 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |           pp8 |         39.97 ± 0.00 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |           pp9 |         74.61 ± 0.86 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |          pp10 |         83.43 ± 0.33 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |          pp11 |         93.02 ± 0.25 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |          pp12 |        101.27 ± 0.33 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |          pp13 |        109.09 ± 0.25 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |          pp14 |        117.88 ± 0.22 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |          pp15 |        125.75 ± 0.50 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |          pp16 |        134.36 ± 0.92 |

build: 0c39f44d (4230)

3090:

$ CUDA_VISIBLE_DEVICES=GPU-6f ./llama-bench --model /mnt/nvme/models/Qwen2.5-Coder-32B-Instruct-Q4_K_M.gguf -p 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16 -10:38:28 [17/17]
1
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
| model                          |       size |     params | backend    | ngl | fa |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | ------------: | -------------------: |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp1 |         37.37 ± 0.14 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp2 |         62.71 ± 0.88 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp3 |         75.65 ± 0.21 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp4 |         82.47 ± 1.55 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp5 |         94.77 ± 1.11 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp6 |         95.37 ± 2.11 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp7 |         99.23 ± 1.04 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp8 |        104.21 ± 1.48 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp9 |        204.45 ± 0.30 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp10 |        226.67 ± 0.35 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp11 |        248.28 ± 0.44 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp12 |        267.58 ± 0.98 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp13 |        291.28 ± 0.94 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp14 |        312.86 ± 0.92 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp15 |        332.97 ± 1.27 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp16 |        355.07 ± 0.80 |

build: 0c39f44d (4230)

$ CUDA_VISIBLE_DEVICES=GPU-6f ./llama-bench --model /mnt/nvme/models/Qwen2.5-Coder-32B-Instruct-Q4_K_M.gguf -p 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16 -10:38:28 [17/17]
1
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
| model                          |       size |     params | backend    | ngl | fa |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | ------------: | -------------------: |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp1 |         37.37 ± 0.14 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp2 |         62.71 ± 0.88 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp3 |         75.65 ± 0.21 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp4 |         82.47 ± 1.55 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp5 |         94.77 ± 1.11 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp6 |         95.37 ± 2.11 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp7 |         99.23 ± 1.04 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp8 |        104.21 ± 1.48 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp9 |        204.45 ± 0.30 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp10 |        226.67 ± 0.35 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp11 |        248.28 ± 0.44 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp12 |        267.58 ± 0.98 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp13 |        291.28 ± 0.94 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp14 |        312.86 ± 0.92 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp15 |        332.97 ± 1.27 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp16 |        355.07 ± 0.80 |

P40:

$ CUDA_VISIBLE_DEVICES=GPU-eb1 ./llama-bench --model /mnt/nvme/models/Qwen2.5-Coder-32B-Instruct-Q4_K_M.gguf -p 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16 -n 0 -ngl 99 -fa 1
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: Tesla P40, compute capability 6.1, VMM: yes
| model                          |       size |     params | backend    | ngl | fa |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | ------------: | -------------------: |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp1 |         11.37 ± 0.01 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp2 |         21.28 ± 0.01 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp3 |         27.17 ± 0.10 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp4 |         29.26 ± 0.02 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp5 |         32.71 ± 0.00 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp6 |         31.50 ± 0.02 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp7 |         34.79 ± 0.00 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp8 |         35.77 ± 0.11 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |           pp9 |         67.21 ± 0.06 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp10 |         74.52 ± 0.05 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp11 |         81.85 ± 0.04 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp12 |         89.06 ± 0.08 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp13 |         96.30 ± 0.09 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp14 |        103.47 ± 0.08 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp15 |        110.66 ± 0.13 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |  1 |          pp16 |        117.79 ± 0.10 |

$ CUDA_VISIBLE_DEVICES=GPU-eb1 ./llama-bench --model /mnt/nvme/models/Qwen2.5-Coder-32B-Instruct-Q4_K_M.gguf -p 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16 -n 0 -ngl 99 -fa 1 -ctk q8_0 -ctv q8_0
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: Tesla P40, compute capability 6.1, VMM: yes
| model                          |       size |     params | backend    | ngl | type_k | type_v | fa |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | -: | ------------: | -------------------: |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |           pp1 |         11.16 ± 0.02 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |           pp2 |         20.99 ± 0.00 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |           pp3 |         26.91 ± 0.01 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |           pp4 |         28.86 ± 0.05 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |           pp5 |         32.39 ± 0.01 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |           pp6 |         33.84 ± 0.02 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |           pp7 |         34.49 ± 0.01 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |           pp8 |         35.54 ± 0.01 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |           pp9 |         66.24 ± 0.32 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |          pp10 |         73.50 ± 0.03 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |          pp11 |         80.64 ± 0.09 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |          pp12 |         87.86 ± 0.09 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |          pp13 |         94.94 ± 0.07 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |          pp14 |        102.07 ± 0.06 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |          pp15 |        109.13 ± 0.05 |
| qwen2 32B Q4_K - Medium        |  18.48 GiB |    32.76 B | CUDA       |  99 |   q8_0 |   q8_0 |  1 |          pp16 |        116.16 ± 0.09 |

build: 0c39f44d (4230)

[ggerganov]

The numbers look OK. Not sure what is causing this slow down. Does adding "top_k": 1 to the requests help? Btw, what is the base line performance for these prompts (i.e. without draft)?

[mostlygeek]

Adding top_k didn't seem to make much difference

$ for model in "qwen-coder-32b-q4-nodraft" "qwen-coder-32b-q4" "qwen-coder-32b-q4-w-ctk"; do 
    for lang in "python" "typescript" "swift"; do
        echo "Generating Snake Game in $lang using $model"
        curl -s --url http://localhost:8080/v1/chat/completions -d "{\"messages\": [{\"role\": \"system\", \"content\": \"you only write code.\"}, {\"role\": \"user\", \"content\": \"write snake game in $lang\"}], \"temperature\": 0.1, \"top_k\": 1, \"model\":\"$model\"}" > /dev/null
        curl -s --url http://localhost:8080/logs | grep 'eval time' | tail -n 2
        echo ""
    done
done

# No Draft Model

Generating Snake Game in python using qwen-coder-32b-q4-nodraft
prompt eval time =     109.70 ms /     6 tokens (   18.28 ms per token,    54.69 tokens per second)
       eval time =   26295.40 ms /   897 tokens (   29.31 ms per token,    34.11 tokens per second)

Generating Snake Game in typescript using qwen-coder-32b-q4-nodraft
prompt eval time =      70.33 ms /     7 tokens (   10.05 ms per token,    99.53 tokens per second)
       eval time =   27249.02 ms /   927 tokens (   29.39 ms per token,    34.02 tokens per second)

Generating Snake Game in swift using qwen-coder-32b-q4-nodraft
prompt eval time =      60.11 ms /     6 tokens (   10.02 ms per token,    99.82 tokens per second)
       eval time =   28842.03 ms /   981 tokens (   29.40 ms per token,    34.01 tokens per second)

# With Draft model 

Generating Snake Game in python using qwen-coder-32b-q4
prompt eval time =      64.66 ms /    23 tokens (    2.81 ms per token,   355.69 tokens per second)
       eval time =   11050.18 ms /   900 tokens (   12.28 ms per token,    81.45 tokens per second)

Generating Snake Game in typescript using qwen-coder-32b-q4
prompt eval time =      68.51 ms /     7 tokens (    9.79 ms per token,   102.17 tokens per second)
       eval time =   15571.32 ms /   928 tokens (   16.78 ms per token,    59.60 tokens per second)

Generating Snake Game in swift using qwen-coder-32b-q4
prompt eval time =      60.99 ms /     6 tokens (   10.17 ms per token,    98.37 tokens per second)
       eval time =   22809.77 ms /   981 tokens (   23.25 ms per token,    43.01 tokens per second)

# Draft model with  --cache-type-k q8_0 --cache-type-v q8_0 

Generating Snake Game in python using qwen-coder-32b-q4-w-ctk
prompt eval time =      52.30 ms /    23 tokens (    2.27 ms per token,   439.79 tokens per second)
       eval time =   13230.13 ms /   900 tokens (   14.70 ms per token,    68.03 tokens per second)

Generating Snake Game in typescript using qwen-coder-32b-q4-w-ctk
prompt eval time =      69.92 ms /     7 tokens (    9.99 ms per token,   100.12 tokens per second)
       eval time =   19563.82 ms /   899 tokens (   21.76 ms per token,    45.95 tokens per second)

Generating Snake Game in swift using qwen-coder-32b-q4-w-ctk
prompt eval time =      60.63 ms /     6 tokens (   10.11 ms per token,    98.96 tokens per second)
       eval time =   26386.14 ms /   957 tokens (   27.57 ms per token,    36.27 tokens per second)

[ggerganov]

I think it should be fixed now - pull the latest #10586 and try again.

[mostlygeek]

Nice! That fixed it.

# No Draft Model
Generating Snake Game in python using qwen-coder-32b-q4-nodraft
prompt eval time =      51.70 ms /    23 tokens (    2.25 ms per token,   444.87 tokens per second)
       eval time =   26287.41 ms /   897 tokens (   29.31 ms per token,    34.12 tokens per second)

Generating Snake Game in typescript using qwen-coder-32b-q4-nodraft
prompt eval time =      69.33 ms /     7 tokens (    9.90 ms per token,   100.97 tokens per second)
       eval time =   27221.19 ms /   927 tokens (   29.36 ms per token,    34.05 tokens per second)

Generating Snake Game in swift using qwen-coder-32b-q4-nodraft
prompt eval time =      59.59 ms /     6 tokens (    9.93 ms per token,   100.70 tokens per second)
       eval time =   28852.84 ms /   981 tokens (   29.41 ms per token,    34.00 tokens per second)

# With Draft model 

Generating Snake Game in python using qwen-coder-32b-q4
prompt eval time =      53.64 ms /    23 tokens (    2.33 ms per token,   428.81 tokens per second)
       eval time =   11044.93 ms /   900 tokens (   12.27 ms per token,    81.49 tokens per second)

Generating Snake Game in typescript using qwen-coder-32b-q4
prompt eval time =      69.26 ms /     7 tokens (    9.89 ms per token,   101.07 tokens per second)
       eval time =   15640.30 ms /   928 tokens (   16.85 ms per token,    59.33 tokens per second)

Generating Snake Game in swift using qwen-coder-32b-q4
prompt eval time =      60.38 ms /     6 tokens (   10.06 ms per token,    99.37 tokens per second)
       eval time =   22865.86 ms /   981 tokens (   23.31 ms per token,    42.90 tokens per second)

# Draft model with  --cache-type-k q8_0 --cache-type-v q8_0 

Generating Snake Game in python using qwen-coder-32b-q4-w-ctk
prompt eval time =      54.38 ms /    23 tokens (    2.36 ms per token,   422.92 tokens per second)
       eval time =   11156.33 ms /   900 tokens (   12.40 ms per token,    80.67 tokens per second)

Generating Snake Game in typescript using qwen-coder-32b-q4-w-ctk
prompt eval time =      68.78 ms /     7 tokens (    9.83 ms per token,   101.78 tokens per second)
       eval time =   16577.13 ms /   885 tokens (   18.73 ms per token,    53.39 tokens per second)

Generating Snake Game in swift using qwen-coder-32b-q4-w-ctk
prompt eval time =      62.71 ms /     6 tokens (   10.45 ms per token,    95.68 tokens per second)
       eval time =   22280.34 ms /  1025 tokens (   21.74 ms per token,    46.00 tokens per second)

[felladrin]

Thank you both for the investigation and resolution!

I've tested the latest changes on Qwen Coder 14B and 32B in several different queries and, in average, on a M1 Max it increased 12 t/s for the 14B model and 5 t/s for the 32B, when using --draft-max 16 --draft-min 4 --draft-p-min 0.4.

[mostlygeek]

fixed in #10586.