Eval bug: Command-A forces full-prompt re-processing due to lack of cache data

[schynce]

Name and Version

version: 5648 (ed52f36)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu

Operating systems

Linux

GGML backends

CUDA

Hardware

3x RTX 3090

Models

Cohere Command-A Q4_K_S

Problem description & steps to reproduce

Requests with the same prompt do not properly utilize the KV cache:

If I request a text completion or chat completion once, and then request the same completion again with the same content, the context is processed from the beginning instead of utilizing the kv cache.

The relevant line in the log:

forcing full prompt re-processing due to lack of cache data (likely due to SWA, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)

Before the "first bad commit", regenerating requests worked as expected.

First Bad Commit

3600cc2 is the first bad commit
commit 3600cc2
Author: Georgi Gerganov ggerganov@gmail.com
Date: Sat May 31 15:57:44 2025 +0300

llama : use n_swa + n_ubatch cells for SWA cache (#13833)

* llama : use n_swa + n_ubatch cells for SWA cache

ggml-ci

* llama : add warning about multi-sqeuence SWA contexts

Relevant log output

./llama-server -m CohereForAI_c4ai-command-a-03-2025-Q4_K_S-00001-of-00002.gguf -c 4096 --temp 0.7 --min_p 0.1 -ctk q8_0 -ctv q8_0 -fa -ngl 999 --jinja

srv  update_slots: all slots are idle
srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 40960, n_keep = 0, n_prompt_tokens = 524
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 524, n_tokens = 524, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 524, n_tokens = 524
slot      release: id  0 | task 0 | stop processing: n_past = 526, truncated = 0
slot print_timing: id  0 | task 0 |
prompt eval time =    3220.10 ms /   524 tokens (    6.15 ms per token,   162.73 tokens per second)
       eval time =     177.28 ms /     3 tokens (   59.09 ms per token,    16.92 tokens per second)
      total time =    3397.38 ms /   527 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 200
srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 4 | processing task
slot update_slots: id  0 | task 4 | new prompt, n_ctx_slot = 40960, n_keep = 0, n_prompt_tokens = 524
slot update_slots: id  0 | task 4 | n_past = 524, cache_tokens.size() = 526, seq_id = 0, pos_min = 0, n_swa = 4096
slot update_slots: id  0 | task 4 | forcing full prompt re-processing due to lack of cache data (likely due to SWA, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 4 | kv cache rm [0, end)
slot update_slots: id  0 | task 4 | prompt processing progress, n_past = 524, n_tokens = 524, progress = 1.000000
slot update_slots: id  0 | task 4 | prompt done, n_past = 524, n_tokens = 524
slot      release: id  0 | task 4 | stop processing: n_past = 527, truncated = 0
slot print_timing: id  0 | task 4 |
prompt eval time =    3233.55 ms /   524 tokens (    6.17 ms per token,   162.05 tokens per second)
       eval time =     239.97 ms /     4 tokens (   59.99 ms per token,    16.67 tokens per second)
      total time =    3473.52 ms /   528 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 200

[ggerganov]

@schynce Could you confirm that #14163 fixes the issue?

[schynce]

@schynce Could you confirm that #14163 fixes the issue?

Thank you!

The commit appears to solve the issue for short prompts. Unfortunately, it has problems with longer prompts.

Here's how it acts now during regeneration with short prompts:

srv  update_slots: all slots are idle
srv  params_from_: Chat format: Content-only                                                                                 
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 40960, n_keep = 0, n_prompt_tokens = 524
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 524, n_tokens = 524, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 524, n_tokens = 524
slot      release: id  0 | task 0 | stop processing: n_past = 526, truncated = 0
slot print_timing: id  0 | task 0 |
prompt eval time =    3270.99 ms /   524 tokens (    6.24 ms per token,   160.20 tokens per second)
       eval time =     171.64 ms /     3 tokens (   57.21 ms per token,    17.48 tokens per second)
      total time =    3442.63 ms /   527 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 200
srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 4 | processing task
slot update_slots: id  0 | task 4 | new prompt, n_ctx_slot = 40960, n_keep = 0, n_prompt_tokens = 524
slot update_slots: id  0 | task 4 | need to evaluate at least 1 token to generate logits, n_past = 524, n_prompt_tokens = 524
slot update_slots: id  0 | task 4 | kv cache rm [523, end)
slot update_slots: id  0 | task 4 | prompt processing progress, n_past = 524, n_tokens = 1, progress = 0.001908
slot update_slots: id  0 | task 4 | prompt done, n_past = 524, n_tokens = 1
slot      release: id  0 | task 4 | stop processing: n_past = 527, truncated = 0
slot print_timing: id  0 | task 4 |
prompt eval time =     108.25 ms /     1 tokens (  108.25 ms per token,     9.24 tokens per second)
       eval time =     245.64 ms /     4 tokens (   61.41 ms per token,    16.28 tokens per second)
      total time =     353.89 ms /     5 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 200
srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 9 | processing task
slot update_slots: id  0 | task 9 | new prompt, n_ctx_slot = 40960, n_keep = 0, n_prompt_tokens = 524
slot update_slots: id  0 | task 9 | need to evaluate at least 1 token to generate logits, n_past = 524, n_prompt_tokens = 524
slot update_slots: id  0 | task 9 | kv cache rm [523, end)
slot update_slots: id  0 | task 9 | prompt processing progress, n_past = 524, n_tokens = 1, progress = 0.001908
slot update_slots: id  0 | task 9 | prompt done, n_past = 524, n_tokens = 1
slot      release: id  0 | task 9 | stop processing: n_past = 526, truncated = 0
slot print_timing: id  0 | task 9 |
prompt eval time =      87.44 ms /     1 tokens (   87.44 ms per token,    11.44 tokens per second)
      eval time =     162.11 ms /     3 tokens (   54.04 ms per token,    18.51 tokens per second)
      total time =     249.55 ms /     4 tokens
srv  update_slots: all slots are idle

With a longer prompt, it still does re-processing:

slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 40960, n_keep = 0, n_prompt_tokens = 13296
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.154031
slot update_slots: id  0 | task 0 | kv cache rm [2048, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 4096, n_tokens = 2048, progress = 0.308063
slot update_slots: id  0 | task 0 | kv cache rm [4096, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 6144, n_tokens = 2048, progress = 0.462094
slot update_slots: id  0 | task 0 | kv cache rm [6144, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 8192, n_tokens = 2048, progress = 0.616125
slot update_slots: id  0 | task 0 | kv cache rm [8192, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 10240, n_tokens = 2048, progress = 0.770156
slot update_slots: id  0 | task 0 | kv cache rm [10240, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 12288, n_tokens = 2048, progress = 0.924188
slot update_slots: id  0 | task 0 | kv cache rm [12288, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 13296, n_tokens = 1008, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 13296, n_tokens = 1008
slot      release: id  0 | task 0 | stop processing: n_past = 13867, truncated = 0
slot print_timing: id  0 | task 0 |
prompt eval time =   86323.45 ms / 13296 tokens (    6.49 ms per token,   154.03 tokens per second)
       eval time =   77753.61 ms /   572 tokens (  135.93 ms per token,     7.36 tokens per second)
      total time =  164077.07 ms / 13868 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /completion 127.0.0.1 200
slot launch_slot_: id  0 | task 579 | processing task
slot update_slots: id  0 | task 579 | new prompt, n_ctx_slot = 40960, n_keep = 0, n_prompt_tokens = 13296
slot update_slots: id  0 | task 579 | n_past = 13296, cache_tokens.size() = 13867, seq_id = 0, pos_min = 9259, n_swa = 4096
slot update_slots: id  0 | task 579 | forcing full prompt re-processing due to lack of cache data (likely due to SWA, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 579 | kv cache rm [0, end)
slot update_slots: id  0 | task 579 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.154031

However, if we use both a short and a long prompt, we run into bigger problems:

srv  update_slots: all slots are idle
srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 40960, n_keep = 0, n_prompt_tokens = 524
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 524, n_tokens = 524, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 524, n_tokens = 524
slot      release: id  0 | task 0 | stop processing: n_past = 526, truncated = 0
slot print_timing: id  0 | task 0 |
prompt eval time =    3487.89 ms /   524 tokens (    6.66 ms per token,   150.23 tokens per second)
       eval time =     227.13 ms /     3 tokens (   75.71 ms per token,    13.21 tokens per second)
      total time =    3715.02 ms /   527 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 200
srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 4 | processing task
slot update_slots: id  0 | task 4 | new prompt, n_ctx_slot = 40960, n_keep = 0, n_prompt_tokens = 524
slot update_slots: id  0 | task 4 | need to evaluate at least 1 token to generate logits, n_past = 524, n_prompt_tokens = 524
slot update_slots: id  0 | task 4 | kv cache rm [523, end)
slot update_slots: id  0 | task 4 | prompt processing progress, n_past = 524, n_tokens = 1, progress = 0.001908
slot update_slots: id  0 | task 4 | prompt done, n_past = 524, n_tokens = 1
slot      release: id  0 | task 4 | stop processing: n_past = 526, truncated = 0
slot print_timing: id  0 | task 4 |
prompt eval time =     114.92 ms /     1 tokens (  114.92 ms per token,     8.70 tokens per second)
       eval time =     235.82 ms /     3 tokens (   78.61 ms per token,    12.72 tokens per second)
      total time =     350.75 ms /     4 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 200
srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 8 | processing task
slot update_slots: id  0 | task 8 | new prompt, n_ctx_slot = 40960, n_keep = 0, n_prompt_tokens = 2775
slot update_slots: id  0 | task 8 | kv cache rm [3, end)
slot update_slots: id  0 | task 8 | prompt processing progress, n_past = 2051, n_tokens = 2048, progress = 0.738018
slot update_slots: id  0 | task 8 | kv cache rm [2051, end)
slot update_slots: id  0 | task 8 | prompt processing progress, n_past = 2775, n_tokens = 724, progress = 0.998919
slot update_slots: id  0 | task 8 | prompt done, n_past = 2775, n_tokens = 724
slot      release: id  0 | task 8 | stop processing: n_past = 2940, truncated = 0
slot print_timing: id  0 | task 8 |
prompt eval time =   17859.01 ms /  2772 tokens (    6.44 ms per token,   155.22 tokens per second)
       eval time =   21292.32 ms /   166 tokens (  128.27 ms per token,     7.80 tokens per second)
      total time =   39151.33 ms /  2938 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 200
srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 176 | processing task
slot update_slots: id  0 | task 176 | new prompt, n_ctx_slot = 40960, n_keep = 0, n_prompt_tokens = 2775
slot update_slots: id  0 | task 176 | need to evaluate at least 1 token to generate logits, n_past = 2775, n_prompt_tokens = 2775
slot update_slots: id  0 | task 176 | kv cache rm [2774, end)
slot update_slots: id  0 | task 176 | prompt processing progress, n_past = 2775, n_tokens = 1, progress = 0.000360
slot update_slots: id  0 | task 176 | prompt done, n_past = 2775, n_tokens = 1
slot      release: id  0 | task 176 | stop processing: n_past = 2959, truncated = 0
slot print_timing: id  0 | task 176 |
prompt eval time =     178.03 ms /     1 tokens (  178.03 ms per token,     5.62 tokens per second)
       eval time =   23477.59 ms /   185 tokens (  126.91 ms per token,     7.88 tokens per second)
      total time =   23655.62 ms /   186 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 200
srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 362 | processing task
slot update_slots: id  0 | task 362 | new prompt, n_ctx_slot = 40960, n_keep = 0, n_prompt_tokens = 7294
slot update_slots: id  0 | task 362 | kv cache rm [3, end)
slot update_slots: id  0 | task 362 | prompt processing progress, n_past = 2051, n_tokens = 2048, progress = 0.280779
slot update_slots: id  0 | task 362 | kv cache rm [2051, end)
slot update_slots: id  0 | task 362 | prompt processing progress, n_past = 4099, n_tokens = 2048, progress = 0.561557
slot update_slots: id  0 | task 362 | kv cache rm [4099, end)
slot update_slots: id  0 | task 362 | prompt processing progress, n_past = 6147, n_tokens = 2048, progress = 0.842336
decode: failed to find a memory slot for batch of size 2048
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 0, n_batch = 1024, ret = 1
decode: failed to find a memory slot for batch of size 1024
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 0, n_batch = 512, ret = 1
decode: failed to find a memory slot for batch of size 512
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 0, n_batch = 256, ret = 1
decode: failed to find a memory slot for batch of size 1792
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 256, n_batch = 1024, ret = 1
decode: failed to find a memory slot for batch of size 1024
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 256, n_batch = 512, ret = 1
decode: failed to find a memory slot for batch of size 512
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 256, n_batch = 256, ret = 1
decode: failed to find a memory slot for batch of size 1536
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 512, n_batch = 1024, ret = 1
decode: failed to find a memory slot for batch of size 1024
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 512, n_batch = 512, ret = 1
decode: failed to find a memory slot for batch of size 512
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 512, n_batch = 256, ret = 1
decode: failed to find a memory slot for batch of size 1280
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 768, n_batch = 1024, ret = 1
decode: failed to find a memory slot for batch of size 1024
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 768, n_batch = 512, ret = 1
decode: failed to find a memory slot for batch of size 512
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 768, n_batch = 256, ret = 1
decode: failed to find a memory slot for batch of size 1024
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 1024, n_batch = 1024, ret = 1
decode: failed to find a memory slot for batch of size 1024
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 1024, n_batch = 512, ret = 1
decode: failed to find a memory slot for batch of size 512
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 1024, n_batch = 256, ret = 1
decode: failed to find a memory slot for batch of size 768
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 1280, n_batch = 1024, ret = 1
decode: failed to find a memory slot for batch of size 768
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 1280, n_batch = 512, ret = 1
decode: failed to find a memory slot for batch of size 512
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 1280, n_batch = 256, ret = 1
decode: failed to find a memory slot for batch of size 512
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 1536, n_batch = 1024, ret = 1
decode: failed to find a memory slot for batch of size 512
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 1536, n_batch = 512, ret = 1
decode: failed to find a memory slot for batch of size 512
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 1536, n_batch = 256, ret = 1
slot update_slots: id  0 | task 362 | kv cache rm [6147, end)
slot update_slots: id  0 | task 362 | prompt processing progress, n_past = 7294, n_tokens = 1147, progress = 0.999589
slot update_slots: id  0 | task 362 | prompt done, n_past = 7294, n_tokens = 1147
decode: failed to find a memory slot for batch of size 1147
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 0, n_batch = 1024, ret = 1
decode: failed to find a memory slot for batch of size 1024
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 0, n_batch = 512, ret = 1
decode: failed to find a memory slot for batch of size 512
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 0, n_batch = 256, ret = 1
decode: failed to find a memory slot for batch of size 891
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 256, n_batch = 1024, ret = 1
decode: failed to find a memory slot for batch of size 891
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 256, n_batch = 512, ret = 1
decode: failed to find a memory slot for batch of size 512
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 256, n_batch = 256, ret = 1
decode: failed to find a memory slot for batch of size 635
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 512, n_batch = 1024, ret = 1
decode: failed to find a memory slot for batch of size 635
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 512, n_batch = 512, ret = 1
decode: failed to find a memory slot for batch of size 512
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 512, n_batch = 256, ret = 1
decode: failed to find a memory slot for batch of size 379
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 768, n_batch = 1024, ret = 1
decode: failed to find a memory slot for batch of size 379
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 768, n_batch = 512, ret = 1
decode: failed to find a memory slot for batch of size 379
srv  update_slots: failed to find free space in the KV cache, retrying with smaller batch size, i = 768, n_batch = 256, ret = 1
slot      release: id  0 | task 362 | stop processing: n_past = 8313, truncated = 0
slot print_timing: id  0 | task 362 |
prompt eval time =   39379.97 ms /  7291 tokens (    5.40 ms per token,   185.14 tokens per second)
       eval time =  138794.29 ms /  1020 tokens (  136.07 ms per token,     7.35 tokens per second)
      total time =  178174.26 ms /  8311 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 200
srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 1386 | processing task
slot update_slots: id  0 | task 1386 | new prompt, n_ctx_slot = 40960, n_keep = 0, n_prompt_tokens = 7294
slot update_slots: id  0 | task 1386 | n_past = 7294, cache_tokens.size() = 8313, seq_id = 0, pos_min = 3958, n_swa = 4096
slot update_slots: id  0 | task 1386 | forcing full prompt re-processing due to lack of cache data (likely due to SWA, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 1386 | kv cache rm [0, end)
slot update_slots: id  0 | task 1386 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.280779
slot update_slots: id  0 | task 1386 | kv cache rm [2048, end)
slot update_slots: id  0 | task 1386 | prompt processing progress, n_past = 4096, n_tokens = 2048, progress = 0.561557
slot update_slots: id  0 | task 1386 | kv cache rm [4096, end)
slot update_slots: id  0 | task 1386 | prompt processing progress, n_past = 6144, n_tokens = 2048, progress = 0.842336
slot update_slots: id  0 | task 1386 | kv cache rm [6144, end)
slot update_slots: id  0 | task 1386 | prompt processing progress, n_past = 7294, n_tokens = 1150, progress = 1.000000
slot update_slots: id  0 | task 1386 | prompt done, n_past = 7294, n_tokens = 1150
slot      release: id  0 | task 1386 | stop processing: n_past = 8362, truncated = 0
slot print_timing: id  0 | task 1386 |
prompt eval time =   48978.65 ms /  7294 tokens (    6.71 ms per token,   148.92 tokens per second)
       eval time =  149185.48 ms /  1069 tokens (  139.56 ms per token,     7.17 tokens per second)
      total time =  198164.13 ms /  8363 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 200

Also, I did some more tests with other commits: I found that the initial bad commit I found with git bisect (3600cc2) actually has the re-processing problem with longer prompts, but not with short prompts that I used while doing the bisect for speed. So, the problem was partly introduced before that.

[ggerganov]

These are both expected:

• With SWA models you can rewind only up to ~n_ubatch tokens without reprocessing, though the exact value will depend on the state of the cache
• The warnings about "retrying with smaller batch size" can be ignored

You can avoid both of these by using --swa-full, but this would require more memory.

[schynce]

These are both expected:

* With SWA models you can rewind only up to `~n_ubatch` tokens without reprocessing, though the exact value will depend on the state of the cache

* The warnings about "retrying with smaller batch size" can be ignored

You can avoid both of these by using --swa-full, but this would require more memory.

Ah, I see! Thank you!