[VLM] Merged multi-modal processor for Pixtral

[Flechman]

This PR aims at implementing the merged multi-modal processor for Pixtral as an effort to contribute to the V1 re-arch for multi-modal models.

Additional changes (by @DarkLight1337 ):

• SImplify mask construction for Pixtral-HF
• Update type annotation for flatten_2d_lists to avoid unnecessary iteration

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[DarkLight1337]

#12767 should make it easier to pass the image token ID

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Flechman.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[DarkLight1337]

Any update on this?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Flechman.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[DarkLight1337]

Evaluation against mmmu_val using lmms_eval:

[PR #14806]
V0:
- mistralai/Pixtral-12B-2409: 0.5044
- mistral-community/pixtral-12b: 0.5056

[PR #12211]
V0:
- mistralai/Pixtral-12B-2409: 0.5044
- mistral-community/pixtral-12b: 0.5056

I'm unable to run the eval on V1 because of CUDA re-initialization error. @youkaichao do you have any idea why this happens?

[ywang96]

@DarkLight1337 It's probably related to spawn. I will eval it with mistral-evals.

[ywang96]

Running into issues on V1 - I'll debug

 ValueError: Attempted to assign 0 + 0 = 0 multimodal tokens to 2 placeholders

[ywang96]

python -m eval.run eval_vllm \                                      
        --model_name mistralai/Pixtral-12B-2409 \
        --url http://0.0.0.0:8000/ \
        --output_dir ~/tmp \
        --eval_name docvqa

V0 main

{
    "anls": 0.8834957820937713
}

V1 main

{
    "anls": 0.8837985481702968
}

V0 this PR

{
    "anls": 0.883626647676123
}

V1 this PR (by overriding in arg_utils)

{
    "anls": 0.8837702579252248
}

Given the results are all close enough, this PR should be good to go! Thanks for the work!

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Flechman.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[DarkLight1337]

python -m eval.run eval_vllm \
    --model_name mistral-community/pixtral-12b \
    --url http://0.0.0.0:8000 \
    --output_dir ~/tmp \
    --eval_name docvqa

V0 main

{
    "anls": 0.8941889853837489
}

V1 main

{
    "anls": 0.8943141800499299
}

V0 this PR

{
    "anls": 0.8938789942826822
}

V1 this PR

{
    "anls": 0.8938797419262041
}