[Frontend] decrease import time of vllm.multimodal

[davidxia]

by changing some modules in vllm/multimodal to lazily import expensive modules like transformers or only importing them for type checkers when not used during runtime.

contributes to #14924

python -c 'import vllm'

seems slightly faster

before (main branch commit 302f3ac)

$ hyperfine 'python -c "import vllm"' --warmup 3 --runs 100 --export-markdown
 out.md
Benchmark 1: python -c "import vllm"
  Time (mean ± σ):      9.367 s ±  0.215 s    [User: 8.951 s, System: 2.028 s]
  Range (min … max):    8.931 s … 10.013 s    100 runs

Command	Mean [s]	Min [s]	Max [s]	Relative
python -c "import vllm"	9.367 ± 0.215	8.931	10.013	1.00

after (my PR commit de28f4f933760b7b53aca164ac8c2d7b5256bf11)

$ hyperfine 'python -c "import vllm"' --warmup 3 --runs 100 --export-markdown out.md
Benchmark 1: python -c "import vllm"
  Time (mean ± σ):      9.196 s ±  0.373 s    [User: 8.758 s, System: 2.065 s]
  Range (min … max):    8.837 s … 12.306 s    100 runs

Command	Mean [s]	Min [s]	Max [s]	Relative
python -c "import vllm"	9.196 ± 0.373	8.837	12.306	1.00

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[aarnphm]

Overall looks good, but can you test out if multimodal request still works?

iirc there are some files like config.py where we have to import eagerly (probably good with this PR, but probably still worth to just to perform a quick check)

[aarnphm]

not in the scope of this PR, but ideally we want to reduce this 8s as much as possible with lazy load

[davidxia]

not in the scope of this PR, but ideally we want to reduce this 8s as much as possible with lazy load

what's "8s"?

can you test out if multimodal request still works?

Do you have an example I should run?

[aarnphm]

what's "8s"?

from your hyperfine run, especially User: 8.758 s. This is just a reference for notes, that's all.

Do you have an example I should run?

There are a few examples in vllm/examples here

[davidxia]

from your hyperfine run, especially User: 8.758 s. This is just a reference for notes, that's all.

ah that's right 😅

There are a few examples in vllm/examples here

thanks, I tried that earlier on a CPU platform with vllm serve llava-hf/llava-1.5-7b-hf. But running that file caused the server to crash after sending a 500 response. Do you know if that example only works on GPU platforms and isn't supported with CPU?

[aarnphm]

Ah, let me perfor a quick test then if you don't have access to GPU

[aarnphm]

Ah, let me perfor a quick test then if you don't have access to GPU

This works with phi 3.5 vision. You can use the diff here

diff.patch

[davidxia]

Ah, let me perfor a quick test then if you don't have access to GPU

This works with phi 3.5 vision. You can use the diff here

diff.patch

thanks, applied your patch

[Copilot]

Pull Request Overview

This PR decreases the import time of vllm.multimodal by lazily loading expensive modules and deferring certain imports to type-checking or local scopes.

• Relocate transformers imports from top-level to type-checking blocks or local function scopes.
• Adjust type annotations for improved runtime performance and maintain consistency across modules.

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
vllm/multimodal/processing.py	Moves heavy transformer imports to type-checking and removes redundant quotes.
vllm/multimodal/parse.py	Shifts direct PIL.Image and BatchFeature imports to local scopes in functions.
vllm/multimodal/inputs.py	Implements LazyLoader for torch and refines type aliases and annotations.

[russellb]

Thanks for taking the piecewise approach. This will be easier to review and merge.

[davidxia]

Thanks for taking the piecewise approach. This will be easier to review and merge.

Of course! LazyLoaded PIL.Image in vllm/multimodal/parse.py. Ready for another review. See diff

[aarnphm]

@davidxia if you can apply this patch
diff.patch Thanks I don't have permission to push to your repo

[davidxia]

@davidxia if you can apply this patch diff.patch Thanks I don't have permission to push to your repo

done, thanks!

[aarnphm]

@hmellor from the readthedocs logs it seem to build succesfully? do you know if there are any issue with this?

[hmellor]

@hmellor from the readthedocs logs it seem to build succesfully? do you know if there are any issue with this?

RTD treats warnings as errors:

/home/docs/checkouts/readthedocs.org/user_builds/vllm/checkouts/18031/docs/source/api/summary.md:88: WARNING: Could not find vllm.multimodal.inputs.MultiModalDataDict [autodoc2.missing]
/home/docs/checkouts/readthedocs.org/user_builds/vllm/checkouts/18031/docs/source/api/summary.md:95: WARNING: Could not find vllm.multimodal.inputs.NestedTensors [autodoc2.missing]

[aarnphm]

ah I see.

@davidxia can you move the docstring in TYPE_CHECKING down to the else block instead? Thanks.

[hmellor]

You don't necessarily have to move it, but something with that name has to exist in the else

[davidxia]

@aarnphm @hmellor I'm trying fix the sphinx warnings. I tried copying the same MultiModalDataDict docstring in the if TYPE_CHECKING into the else. But when I run python -m sphinx -T -W --keep-going -b html -d _build/doctrees -D language=en . out/html locally I still see the same warnings. Any ideas?

[aarnphm]

MultiModalDataDict docstring in the if TYPE_CHECKING into the else. But when I run python -m sphinx -T -W --keep-going -b html -d _build/doctrees -D language=en . out/html locally I still see the same warnings. Any ideas?

probably better to keep previous change, but update the annotations to string instead

i.e:

if TYPE_CHECKING:
  import torch
 
HfImageItem: TypeAlias = Union[Image, np.ndarray, "torch.Tensor"]
"""docstring as before"""

[aarnphm]

s/torch.Tensor/"torch.Tensor"