Fix default image_rows and image_cols initialization in Idefics3 and SmolVLM processors

[MilkClouds]

What does this PR do?

This PR fixes a bug in the default initialization of image_rows and image_cols in the Idefics3 and SmolVLM processors.

Problem

The original code incorrectly used len(text) to determine the size of the default image_rows and image_cols lists:

image_rows = inputs.pop("rows", [[0] * len(text)])
image_cols = inputs.pop("cols", [[0] * len(text)])

This creates a single list with length equal to the number of text samples in the batch, which is incorrect. The correct behavior should create a list for each sample, where each inner list has length equal to the number of image tokens in that specific sample.

Solution

Changed the default initialization to count image tokens per sample:

image_rows = inputs.pop("rows", [[0] * sample.count(self.image_token) for sample in text])
image_cols = inputs.pop("cols", [[0] * sample.count(self.image_token) for sample in text])

This ensures that:

1. Each sample in the batch gets its own list
2. Each list has the correct length matching the number of image tokens in that sample

Impact

This fix affects:

• Idefics3Processor in src/transformers/models/idefics3/processing_idefics3.py
• SmolVLMProcessor in src/transformers/models/smolvlm/processing_smolvlm.py

The bug would have caused issues when processing batches with multiple samples or samples with multiple images, as the default values would have incorrect dimensions.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@zucchini-nlp (multimodal models)

[MilkClouds]

p.s. this issue seems to be ignored before because return_row_col_info was always given to True to ImageProcessor! I've met error raised when I used non-SmolVLMImageProcessor for image processor of SmolVLMProcesor.

[MilkClouds]

For reviewer's information: this PR is ready for review!

[yonigozlan]

Good catch! Thanks for fixing. Suggested a minor improvement, but overall LGTM!

[yonigozlan]

You can use n_images_in_text defined above here

[MilkClouds]

I changed the lines as you said!

[yonigozlan]

Same here, you can define image_rows and image_cols before the call to expand_text_with_image_tokens using n_images_in_text

[MilkClouds]

Same here, I changed the lines as you said

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: idefics3, smolvlm

[yonigozlan]

Perfect thanks! I'll merge once the CI passes

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.