CUDA error: out of memory

[SunixLiu]

I successfully installed vLLM in WSL2, when I was trying to run the sample code, I got error info like this:

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="/mnt/d/github/text-generation-webui/models/facebook_opt-125m")

outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

INFO 06-21 21:40:02 llm_engine.py:59] Initializing an LLM engine with config: model='/mnt/d/github/text-generation-webui/models/facebook_opt-125m', dtype=torch.float16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=1, seed=0)
INFO 06-21 21:40:12 llm_engine.py:128] # GPU blocks: 37375, # CPU blocks: 7281
Traceback (most recent call last):
File "/mnt/d/01Projects/vllm/prac_1.py", line 11, in
llm = LLM(model="/mnt/d/github/text-generation-webui/models/facebook_opt-125m")
File "/mnt/d/github/vllm/vllm/entrypoints/llm.py", line 55, in init
self.llm_engine = LLMEngine.from_engine_args(engine_args)
File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 145, in from_engine_args
engine = cls(*engine_configs, distributed_init_method, devices,
File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 102, in init
self._init_cache()
File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 134, in _init_cache
self._run_workers("init_cache_engine", cache_config=self.cache_config)
File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 307, in _run_workers
output = executor(*args, **kwargs)
File "/mnt/d/github/vllm/vllm/worker/worker.py", line 126, in init_cache_engine
self.cache_engine = CacheEngine(
File "/mnt/d/github/vllm/vllm/worker/cache_engine.py", line 41, in init
self.cpu_cache = self.allocate_cpu_cache()
File "/mnt/d/github/vllm/vllm/worker/cache_engine.py", line 89, in allocate_cpu_cache
key_blocks = torch.empty(
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Python: 3.10.11
GPU: RTX 3090 24G
Linux: WSL2, Ubuntu 20.04.6 LTS
Can anyone help to answer this?

[zhuohan123]

The OPT-125M is very small and this cannot happen. While I cannot reproduce the exact error on my side, when you initialize the LLM class, can you try to add the following argument: gpu_memory_utilization=0.80 or set the utilization to an even lower number? The default utilization upper bound is 0.90.

Additionally, can you set CUDA_LAUNCH_BLOCKING=1 to see which exact lines causes the error?

[SunixLiu]

I added two lines:

import os
os.environ ['CUDA_LAUNCH_BLOCKING'] ='1'

and this gpu_memory_utilization=0.50 .

but it seems the error info is the same:

INFO 06-21 22:53:33 llm_engine.py:59] Initializing an LLM engine with config: model='/mnt/d/github/text-generation-webui/models/facebook_opt-125m', dtype=torch.float16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=1, seed=0)
INFO 06-21 22:53:41 llm_engine.py:128] # GPU blocks: 19899, # CPU blocks: 7281
Traceback (most recent call last):
File "/mnt/d/01Projects/vllm/prac_1.py", line 14, in
llm = LLM(model="/mnt/d/github/text-generation-webui/models/facebook_opt-125m",gpu_memory_utilization=0.50)
File "/mnt/d/github/vllm/vllm/entrypoints/llm.py", line 55, in init
self.llm_engine = LLMEngine.from_engine_args(engine_args)
File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 145, in from_engine_args
engine = cls(*engine_configs, distributed_init_method, devices,
File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 102, in init
self._init_cache()
File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 134, in _init_cache
self._run_workers("init_cache_engine", cache_config=self.cache_config)
File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 307, in _run_workers
output = executor(*args, **kwargs)
File "/mnt/d/github/vllm/vllm/worker/worker.py", line 126, in init_cache_engine
self.cache_engine = CacheEngine(
File "/mnt/d/github/vllm/vllm/worker/cache_engine.py", line 41, in init
self.cpu_cache = self.allocate_cpu_cache()
File "/mnt/d/github/vllm/vllm/worker/cache_engine.py", line 89, in allocate_cpu_cache
key_blocks = torch.empty(
RuntimeError: CUDA error: out of memory
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Btw, the text-generation-webui can load the model successfully.

[zhuohan123]

Just to make sure it's not the model's issue, can you run facebook/opt-125m model?

[ZQ-Dev8]

I'm running into the same issue. I'm on Ubuntu WSL2, CUDA 11.8, and RTX 2080Ti, which should easily be able to accommodate opt-125m. I've tried other small models, such as cerebras/Cerebras-GPT-111M and distilgpt2, but those give different errors (probably due to architecture incompatibility?).

Edit: I just tried EleutherAI/pythia-70M-deduped and it too gives the OOM error.

I've confirmed I can run opt-125m and pythia-70m via standard transformers library...any other ideas what might be causing the issue @zhuohan123 @WoosukKwon ?

[SunixLiu]

Just to make sure it's not the model's issue, can you run facebook/opt-125m model?

Yes, I can use text-generation-webui to load facebook/opt-125m and get it work.

[AlpinDale]

Currently having this issue as well. The CUDA_VISIBLE_DEVICES environment variable has no effect either, and it only loads the models to GPU 0. I'm running on A100s but still get OOM with 125m OPT. There's something fatally wrong with this repo.

EDIT: CUDA_VISIBLE_DEVICES seems to work just fine, but the 125m model is using over 37GB of memory.

[WoosukKwon]

@dcruiz01 @SunixLiu @AlpinDale vLLM is designed to take almost all of your GPU memory. Could you double-check your GPU is not used by other processes when using vLLM?

[AlpinDale]

@dcruiz01 @SunixLiu @AlpinDale vLLM is designed to take almost all of your GPU memory. Could you double-check your GPU is not used by other processes when using vLLM?

Thanks, I think I understand now. A somewhat related question - how is multi-GPU handled? If I load a bigger model, will it split across the available GPUs? Also, if I want more memory to accommodate more users on a single, smaller model, can I have the process use multiple GPUs at once?

[WoosukKwon]

@AlpinDale Good question. You can use the tensor_parallel_size argument for multi-GPU inference.
First, initialize your Ray cluster by executing

$ ray start --head

Then, use the tensor_parallel_size argument in the LLM class:

llm = LLM(model=<your model>, tensor_parallel_size=2)  # Inference with 2 GPUs 

[SunixLiu]

@dcruiz01 @SunixLiu @AlpinDale vLLM is designed to take almost all of your GPU memory. Could you double-check your GPU is not used by other processes when using vLLM?

I'm pretty sure there's no other process was using my GPU, and here's the screenshot, you can find a burst in both GPU 3D and VRAM, that happended when the code was trying to load the model.

[ZQ-Dev8]

I'm seeing the same as @SunixLiu. Nothing else is happening on my GPU. Running the code at the top of this issue spikes the GPU and then crashes.

[ZQ-Dev8]

Any new developments here? @zhuohan123

[zhuohan123]

@SunixLiu @dcruiz01 We believe this is actually a bug. We are not sure whether this is caused by WSL or vLLM. We don't have a Windows environment with GPUs. How about we set up a 30-min meeting to pair-debug on your environments? Can you send me an email at zhuohan[at]berkeley.edu to set up a call? Thanks!