[Bugfix] validate urls object for multimodal content parts

[gcalmettes]

TLDR: This PR prevents a 500 Internal Server Error when the url of a multimodal content part object is directly provided as string.

---

When multimodal content is sent to models (e.g.: images for vision models), the multimodal content is sent as content part of the user message:
Multiple syntax format for the _url fields are accepted:

• nested url block as <type>_url value, e.g.:

{
    "type": "image_url",
    "image_url": {
        "url": "https://cms.mistral.ai/assets/1977a3f1-eae3-4eed-b2c7-e1701c4692ed.png?width=994&height=525"
    }
}

• string as <type>_url value if no type key is present (this is undocumented as noted here. This method is only use as fallback if there is no type key), e.g.:

{
    "image_url": "https://cms.mistral.ai/assets/1977a3f1-eae3-4eed-b2c7-e1701c4692ed.png?width=994&height=525"
}

However, currently, if the type field is present, there is no validation of the <type>_url object, even though the urls strings are retrieved using a double get syntax.

As a result, a 500 Internal Server Error will be sent if one send a request like below, instead of a 400 BadRequest error with useful message to the user.

from openai import OpenAI

BASE_URL = "http://localhost:8003/v1"
API_KEY = "not-used"

client = OpenAI(
    base_url=BASE_URL,
    api_key=API_KEY,
)

response = client.chat.completions.create(
    model="mistral-small-3.1",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Extract the information from this image and summarize it as markdown."},
                {
                    "type": "image_url",
                    "image_url": "https://cms.mistral.ai/assets/1977a3f1-eae3-4eed-b2c7-e1701c4692ed.png?width=994&height=525"
                },
            ],
        }
    ]
)

client log:

 line 1057, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.InternalServerError: Internal Server Error

server logs:

INFO:     172.17.0.1:47080 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/prometheus_fastapi_instrumentator/middleware.py", line 177, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/prometheus_fastapi_instrumentator/middleware.py", line 175, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 714, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 734, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/utils.py", line 63, in wrapper
    return handler_task.result()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/utils.py", line 85, in wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 476, in create_chat_completion
    generator = await handler.create_chat_completion(request, raw_request)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 183, in create_chat_completion
    ) = await self._preprocess_chat(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_engine.py", line 391, in _preprocess_chat
    conversation, mm_data_future = parse_chat_messages_futures(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/chat_utils.py", line 1139, in parse_chat_messages_futures
    sub_messages = _parse_chat_message_content(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/chat_utils.py", line 1067, in _parse_chat_message_content
    result = _parse_chat_message_content_parts(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/chat_utils.py", line 967, in _parse_chat_message_content_parts
    parse_res = _parse_chat_message_content_part(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/chat_utils.py", line 1005, in _parse_chat_message_content_part
    part_type, content = _parse_chat_message_content_mm_part(part)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/chat_utils.py", line 914, in _parse_chat_message_content_mm_part
    content = MM_PARSER_MAP[part_type](part)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/chat_utils.py", line 879, in <lambda>
    lambda part: _ImageParser(part).get("image_url", {}).get("url", ""),
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'get'

With this PR

 line 1057, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': "1 validation error for ChatCompletionContentPartImageParam\nimage_url\n  Input should be a valid dictionary [type=dict_type, input_value='https://cms.mistral.ai/a...ng?width=994&height=525', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.11/v/dict_type", 'type': 'BadRequestError', 'param': None, 'code': 400}

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[DarkLight1337]

Thanks for fixing - can you add a regression test for this?

[DarkLight1337]

Thanks!

[gcalmettes]

added regression tests for the 3 modalities concerned (video, image, audio)

[gcalmettes]

I rebased from main to retrigger the CI (arn permissions error)

[2025-04-22T17:48:41Z] An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:sts::936637512419:assumed-role/bk-gpu-1-queue-ci-Role/i-0fcddd2c63116903c is not authorized to perform: secretsmanager:GetSecretValue on resource: ci_hf_token because no identity-based policy allows the secretsmanager:GetSecretValue action
[2025-04-22T17:48:42Z] :alert: Elastic CI Stack environment hook failed
[2025-04-22T17:48:42Z] :alert: Elastic CI Stack environment hook failed
[2025-04-22T17:48:42Z] 🚨 Error: Error setting up job executor: The global environment hook exited with status 53

[gcalmettes]

all tests passed @DarkLight1337 🙏