Image token is different

[Yiting-shen2001]

I try to execute the llama model, and update the version of transformer.
But it seem to ask me to input three image.
I would like to inquire about possible issues and solutions.
Thank you so much!

Traceback (most recent call last):
  File "C:\Users\user\Desktop\XXXX\ComfyUI_windows_portable\ComfyUI\execution.py", line 323, in execute
    output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\Desktop\XXXX\ComfyUI_windows_portable\ComfyUI\execution.py", line 198, in get_output_data
    return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\Desktop\XXXX\ComfyUI_windows_portable\ComfyUI\execution.py", line 169, in _map_node_over_list
    process_inputs(input_dict, i)
  File "C:\Users\user\Desktop\XXXX\ComfyUI_windows_portable\ComfyUI\execution.py", line 158, in process_inputs
    results.append(getattr(obj, func)(**inputs))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\Desktop\XXXX\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-PixtralLlamaMolmoVision\nodes.py", line 281, in generate_text
    inputs = llama_vision_model['processor'](images=image_list, text=final_prompt, return_tensors="pt").to(device)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\Desktop\XXXX\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\mllama\processing_mllama.py", line 309, in __call__
    raise ValueError(
ValueError: The number of image token (3) should be the same as in the number of provided images (1)

[SeanScripts]

What prompt did you use? The number of image tokens you enter in the prompt has to be the same as the number of input images, so make sure you're only using the image token once if you're using a single image as input.

…

On Sat, Jan 11, 2025, 5:52 AM Yiting-shen2001 ***@***.***> wrote: I try to execute the llama model, and update the version of transformer. But it seem to ask me to input three image. I would like to inquire about possible issues and solutions. Thank you so much! Traceback (most recent call last): File "C:\Users\user\Desktop\XXXX\ComfyUI_windows_portable\ComfyUI\execution.py", line 323, in execute output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\user\Desktop\XXXX\ComfyUI_windows_portable\ComfyUI\execution.py", line 198, in get_output_data return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\user\Desktop\XXXX\ComfyUI_windows_portable\ComfyUI\execution.py", line 169, in _map_node_over_list process_inputs(input_dict, i) File "C:\Users\user\Desktop\XXXX\ComfyUI_windows_portable\ComfyUI\execution.py", line 158, in process_inputs results.append(getattr(obj, func)(**inputs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\user\Desktop\XXXX\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-PixtralLlamaMolmoVision\nodes.py", line 281, in generate_text inputs = llama_vision_model['processor'](images=image_list, text=final_prompt, return_tensors="pt").to(device) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\user\Desktop\XXXX\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\mllama\processing_mllama.py", line 309, in __call__ raise ValueError( ValueError: The number of image token (3) should be the same as in the number of provided images (1) — Reply to this email directly, view it on GitHub <#19>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APK3JM7HHK6HCPSVRQQ2OS32KEO27AVCNFSM6AAAAABU772SMWVHI2DSMVQWIX3LMV43ASLTON2WKOZSG44DCOBZGEZDQNA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jervenclark]

I also have the same experience running the sample workflow.

The prompt:

<|begin_of_text|>
<|start_header_id|>user<|end_header_id|>

<|image|>Create an approximate bounding box containing the text in this image. Your response should only include the coordinates of the upper-left and lower-right corners of the bounding box relative to the image size, e.g. [(0.0, 0.0), (1.0, 1.0)] is a bounding box that covers the entire image.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

The bounding box coordinates of the text are: [(

Then I tried batching images because I thought maybe for some reason it's expecting 3 images even if there is only one <|image|> token on the prompt but now the error has changed to The number of image token (6) should be the same as in the number of provided images (4).

This scales to the number of images I add to the batch, if there are 3 images then it has 5 tokens, etc.

If the prompt is blank, it's expecting n+1 images.
I hope this helps triangulate the issue. If you need, I can test more cases.

# ComfyUI Error Report
## Error Details
- **Node ID:** 49
- **Node Type:** LlamaVisionGenerateText
- **Exception Type:** ValueError
- **Exception Message:** The number of image token (5) should be the same as in the number of provided images (4)
## Stack Trace

  File "/stable-diffusion/execution.py", line 323, in execute
    output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)

  File "/stable-diffusion/execution.py", line 198, in get_output_data
    return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)

  File "/stable-diffusion/execution.py", line 169, in _map_node_over_list
    process_inputs(input_dict, i)

  File "/stable-diffusion/execution.py", line 158, in process_inputs
    results.append(getattr(obj, func)(**inputs))

  File "/stable-diffusion/custom_nodes/ComfyUI-PixtralLlamaMolmoVision/nodes.py", line 281, in generate_text

[SeanScripts]

Ah, sorry, I didn't notice that you were running the Llama Vision model. I had previously updated these nodes to allow the user to enter a regular prompt instead of needing to include the special tokens in the prompt like this. Including them is what is overcounting the number of image tokens. Also, with this model, there isn't support for images being added in different places in the prompt like for Pixtral, so the image tokens are just added to the beginning of the prompt automatically. I made a note of this in the readme, but I might not have made it clear enough:
"The Llama and Molmo models will add the images to the beginning of the prompt automatically, and while they do support processing multiple images at once, they don't support including multiple images in different places in the prompt like [Pixtral]"

I notice the workflow images/examples I have in here are from an older version where you needed the special tokens like this, I forgot to update them... I haven't had time to work on this project in a while, sorry.
If you enter this as the prompt (same thing but without the special tokens), then it should run.

Create an approximate bounding box containing the {object} in this image. Your response should only include the coordinates of the upper-left and lower-right corners of the bounding box relative to the image size, e.g. [(0.0, 0.0), (1.0, 1.0)] is a bounding box that covers the entire image.

The bounding box coordinates of the {object} are:

However, I will note that this bounding box example was just something I was testing to see if the model could do it. Llama Vision is actually very bad at this particular task.

I've updated the example image and workflow. In the future I may add another field to the generate text nodes for "start reply with" since it would be helpful for this type of response that expects particular formatting.