[Benchmarks] Include image data when ShareGPT4V dataset is used.

[huachenheli]

Summary:
Currently when ShareGPT4V dataset is used, the image data is not passed to the vLLM server correctly. This PR fixes the issue.

Test Plan:
vLLM:

python -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-VL-7B-Instruct --port 8001 --host 0.0.0.0 --dtype bfloat16 --limit-mm-per-prompt '{"image": 1, "video":1}' --allowed-local-media-path /home/huachenheli/datasets/COCO/train2017

Request sender:

python benchmarks/benchmark_serving.py --backend openai-chat --model Qwen/Qwen2.5-VL-7B-Instruct --dataset-name sharegpt --dataset-path ../ShareGPT4V/sharegpt4v_instruct_gpt4-vision_cap100k_COCO_only.json --num-prompts 10 --port 8001 --save-result --result-dir ~/vllm_benchmark_results --save-detailed --endpoint /v1/chat/completion

Sample config:

[
    {
        "id": "000000000009",
        "image": "/home/huachenheli/datasets/COCO/train2017/000000000009.jpg",
        "conversations": [
            {
                "from": "human",
                "value": "What do you see happening in this image?\n<image>"
            },
            {
                "from": "gpt",
                "value": "In the center of the image, a vibrant blue lunch tray holds four containers, each brimming with a variety of food items. The containers, two in pink and two in yellow, are arranged in a 2x2 grid.\n\nIn the top left pink container, a slice of bread rests, lightly spread with butter and sprinkled with a handful of almonds. The bread is cut into a rectangle, and the almonds are scattered across its buttery surface.\n\nAdjacent to it in the top right corner, another pink container houses a mix of fruit. Sliced apples with their fresh white interiors exposed share the space with juicy chunks of pineapple. The colors of the apple slices and pineapple chunks contrast beautifully against the pink container.\n\nBelow these, in the bottom left corner of the tray, a yellow container holds a single meatball alongside some broccoli. The meatball, round and browned, sits next to the vibrant green broccoli florets.\n\nFinally, in the bottom right yellow container, there's a sweet treat - a chocolate chip cookie. The golden-brown cookie is dotted with chocolate chips, their dark color standing out against the cookie's lighter surface.\n\nThe arrangement of these containers on the blue tray creates a visually appealing and balanced meal, with each component neatly separated yet part of a cohesive whole."
            }
        ]
    },
    ...
]

Reviewers:

@DarkLight1337 @ywang96 @Isotr0py

Subscribers:

Tasks:

Tags:

Purpose

Test Plan

Test Result

(Optional) Documentation Update

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Code Review

This pull request correctly adds support for image data in the ShareGPTDataset for benchmarking. The changes are straightforward and align with the PR's goal. I've identified one potential issue related to handling null or empty image paths in the dataset, which could cause the benchmark to crash. I've included a suggestion to make the image processing more robust.

[DarkLight1337]

Thanks, this looks reasonable to me but I'll have @ywang96 take a look as well

[Isotr0py]

Look reasonable too. Just a nit.

[ywang96]

LGTM! On a side note, we should probably start removing things from benchmarks folder to reduce confusion 😅

[huachenheli]

LGTM! On a side note, we should probably start removing things from benchmarks folder to reduce confusion 😅

SG. I was about to ask why we have two LOL