IDEFICS2 integration test hits OOM on CI

[amyeroberts]

System Info

Running on main from commit 44f6fdd - c.f. CI runs for this PR: #32275

Example run: https://github.com/huggingface/transformers/actions/runs/10134513823/job/28020840326

Opening an issue to keep track

Who can help?

@amyeroberts

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Push a commit with the message [run_slow] idefics2 on a PR with the label run-slow to trigger

Expected behavior

It doesn't run out of mem - the 4 bit integration test might be enough to test however