Misc. bug: 10 Image maximum?

[Jay4242]

Name and Version

$ llama-server --version
version: 5627 (3678b83)
built with cc (Ubuntu 14.2.0-19ubuntu2) 14.2.0 for x86_64-linux-gnu

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server --mmproj /home/{username}/Downloads/models/llama.cpp/bartowski/google_gemma-3-4b-it-GGUF/mmproj-google_gemma-3-4b-it-bf16.gguf -m /home/{username}/Downloads/models/llama.cpp/bartowski/google_gemma-3-4b-it-GGUF/google_gemma-3-4b-it-Q8_0.gguf -t 28 --port 9090 --host 0.0.0.0 -c 0

Problem description & steps to reproduce

To reproduce send 11 images to the server. It's giving me a 500 error on the 11th.
10 images seem to work as expected.
It seemed to allow for many more images (20+) when llama-server was first made multimodal. I'm not sure when this changed.

First Bad Commit

No response

Relevant log output

failed to decode image
failed to decode image                                                                                               decode: failed to find a memory slot for batch of size 256                                                           
srv  process_chun: image processed in 31744 ms                                                                       
mtmd_helper_eval failed with status 1slot update_slots: id  0 | task 840 | failed to process image, res = 1          
slot      release: id  0 | task 840 | stop processing: n_past = 2634, truncated = 0                                  
srv    send_error: task id = 840, error: failed to process image

[ikcikoR]

Happens to me as well, tho I can get 11 images, then 12th fails to process. Changing context size doesn't help. I'm using SillyTavern as the frontend through OpenAI-compatible mode since that's the only way it supports multimodal LLMs tho I don't think that changes anything here.

srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 117 | processing task
slot update_slots: id  0 | task 117 | new prompt, n_ctx_slot = 24576, n_keep = 0, n_prompt_tokens = 5970
slot update_slots: id  0 | task 117 | kv cache rm [5695, end)
srv  process_chun: processing image...
encoding image slice...
image slice encoded in 426 ms
decoding image batch 1/1, n_tokens_batch = 256
failed to decode image
failed to decode image
decode: failed to find a memory slot for batch of size 256
srv  process_chun: image processed in 426 ms
mtmd_helper_eval failed with status 1slot update_slots: id  0 | task 117 | failed to process image, res = 1
slot      release: id  0 | task 117 | stop processing: n_past = 5695, truncated = 0
srv    send_error: task id = 117, error: failed to process image
srv  update_slots: no tokens to decode
srv  update_slots: all slots are idle
srv  cancel_tasks: cancel task, id_task = 117
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /chat/completions 127.0.0.1 200

[ikcikoR]

Huh, nevermind, even deleting earlier images from context somehow doesn't make the problem go away now

[steampunque]

I think its something to do with SWA. I did a quick test with my downstream server with gemma 12b with SWA enabled and it dies on the tenth image with decode: failed to find a memory slot for batch of size 128, doesn't matter how big the context is. Set --swa-full and it handled 15 images no issue, most likely would continue handling more until it runs out of context (which it will do quickly with --swa-full)

[Jay4242]

Thanks to both of you for looking into it! Adding --swa-full did seem to solve it. llama-server was able to process 24 images as a test.

Sorry if this was me not understanding SWA.

Thanks again!

[steampunque]

Thanks to both of you for looking into it! Adding --swa-full did seem to solve it. llama-server was able to process 24 images as a test.

Sorry if this was me not understanding SWA.

Thanks again!

I think this is a legit bug. It should be working without the need for --swa-full. Some kind of fragmentation going on in the KV management. Without --swa-full you should be able to input a huge amount of images with the much larger context.

[ggerganov]

Could you check if #14119 fixes the issue?

[steampunque]

Could you check if #14119 fixes the issue?

Looks good with my downstream server at 15 images with gemma3 12b. Test prompt I use to check if its reading them in right:

fifteen.txt: Fifteen images follow.  Which of them are the same?: 1: @boxes_512.png 2: @boxes_512.png 3: @boxes_512.png 4: @boxes_512.png 5: @bird_cropped.png 6: @boxes_512.png 7: @boxes_512.png 8: @boxes_512.png 9: @boxes_512.png 10: @bird_cropped.png 11: @bandwagon_512.png 12: @bird_cropped.png 13: @bird_cropped.png 14: @bandwagon_512.png 15: @boxes_512.png

bash-5.1$ lm fifteen.txt
Here's the breakdown of which images are the same:

• Images 1, 2, 3, 4, 6, 7, 8, 9, and 15 are all identical. They all show the same arrangement of squares.
• Images 5, 10, 12, and 13 are identical. They all show the same image of a bird on a pedestal.
• Images 11 and 14 are identical. They both show the same page of text.

So, there are two distinct groups of identical images.

Mostly correct except for final conclusion from gemma 12b but looking quite good overall. I think this verifies attention correct as the 15 images should have pushed outside the sliding window boundary. Thanks for quick resolution of issue!