[Feature]: Reduce vLLM's import time

[simon-mo]

🚀 The feature, motivation and pitch

It takes 6s to print a version, likely because vLLM initialize the CUDA context through import

time vllm --version
INFO 03-17 04:53:22 [__init__.py:256] Automatically detected platform cuda.
0.7.4.dev497+ga73e183e

real    0m4.729s
user    0m5.921s
sys     0m6.833s

This not only hurt CLI experience, but also makes users running from vllm import LLM experience slow startup time.

Please help us investigate this and make import time computation as lazy as possible so a simple vllm --version can be ran fast.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[lengrongfu]

I add print import time, but not time-consuming.

[youkaichao]

likely because vLLM initialize the CUDA context through import

not initialize the cuda context, but it detects the current platform.

to reduce the import time, we should defer some imports for vllm.platforms.current_platform, i think that would be the most time-consuing part.

[aarnphm]

quickly chime in. Here's the log trace for the first initialisation

INFO 03-22 08:17:39 [__init__.py:256] Automatically detected platform cuda.
INFO 03-22 08:17:40 [api_server.py:981] vLLM API server version 0.7.3.dev768+gc0efdd65
INFO 03-22 08:17:40 [api_server.py:982] args: ....
INFO 03-22 08:17:40 [config.py:2589] Downcasting torch.float32 to torch.bfloat16.
INFO 03-22 08:17:47 [config.py:585] This model supports multiple tasks: {'reward', 'embed', 'classify', 'generate', 'score'}. Defaulting to 'generate'.
INFO 03-22 08:17:47 [config.py:1687] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 03-22 08:17:52 [__init__.py:256] Automatically detected platform cuda.
INFO 03-22 08:17:54 [core.py:54] Initializing a V1 LLM engine (v0.7.3.dev768+gc0efdd65)

We are checking the platform multiple times here, probably best to lazy init this and called it only once?

I also think that we should try to lazy import most imports initially, so that import vllm should be quick

import sys
import timeit
def measure_import(lib_name):
  stmt = f"import sys; sys.modules.pop('{lib_name}', None); import {lib_name}"
  return timeit.timeit(stmt, number=1)

measure_import('vllm')

yields 3.4815449090019683

Here's the cprofile file: import.prof.zip

import transformers eagerly take 2 seconds!

I propose that we can lazy setup all toplevel modules from vllm, to make sure that we don't try to import all these module eagerly.

[Chen-0210]

likely because vLLM initialize the CUDA context through import

not initialize the cuda context, but it detects the current platform.

to reduce the import time, we should defer some imports for vllm.platforms.current_platform, i think that would be the most time-consuing part.

Yes, I noticed that. Besides platform detection, it also includes imports of heavy modules like transformers.

[Chen-0210]

Thanks @aarnphm! I’ve already covered most of it — But I need to tidy up a few parts.

[davidxia]

Thanks @aarnphm! I’ve already covered most of it — But I need to tidy up a few parts.

for visibility, @Chen-0210's PR is #15128