RuntimeError: CUDA error: no kernel image is available for execution on the device

[rookielyb]

Error:
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

nvcc -V
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_May__3_18:49:52_PDT_2022
Cuda compilation tools, release 11.7, V11.7.64
Build cuda_11.7.r11.7/compiler.31294372_0

conda list:
cudatoolkit-dev 11.7.0
cudatoolkit 11.7.0
torch 2.0.1+cu117

nvidia-smi
A100 80G
NVIDIA-SMI 470.141.03 Driver Version: 470.141.03 CUDA Version: 11.4

how to solve this problem? thanks!

[rookielyb]

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.1 LTS
Release: 20.04
Codename: focal

cat /proc/version
Linux version 5.4.0-126-generic (buildd@lcy02-amd64-072) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)) #142 SMP Fri Aug 26 12:12:57 UTC 2022

[LiuXiaoxuanPKU]

could you check that the problem still exits after rebuilding the repo (pip install -e .)?

[rookielyb]

could you check that the problem still exits after rebuilding the repo (pip install -e .)?
pip install -e . :
Building wheels for collected packages: vllm
Building editable for vllm (pyproject.toml) ... done
Created wheel for vllm: filename=vllm-0.1.2-0.editable-cp310-cp310-linux_x86_64.whl size=8465 sha256=8154890edc8a5b3b0100d83308d973ec25ecee72b02d479023542312fee2fd1d
Stored in directory: /tmp/pip-ephem-wheel-cache-vg3isffo/wheels/33/fc/d6/f27b3ac96c14477426ab8fd6d5573e139cf29c857e206d16a3
Successfully built vllm
Installing collected packages: vllm
Attempting uninstall: vllm
Found existing installation: vllm 0.1.2
Uninstalling vllm-0.1.2:
Successfully uninstalled vllm-0.1.2
Successfully installed vllm-0.1.2

CUDA_VISIBLE_DEVICES=0 python offline_inference.py:
File "/home/anaconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in call_impl
return forward_call(*args, **kwargs)
File "/cg/vllm/vllm/model_executor/models/opt.py", line 102, in forward
output, _ = self.out_proj(attn_output)
File "/home/anaconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in call_impl
return forward_call(*args, **kwargs)
File "/cg/vllm/vllm/model_executor/parallel_utils/tensor_parallel/layers.py", line 443, in forward
output = output + self.bias if self.bias is not None else output
RuntimeError: CUDA error: no kernel image is available for execution on the device
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Processed prompts: 0%| | 0/4 [00:00<?, ?it/s]

still not resolved

[YHPeter]

I have the same issue. Same nvcc/driver both 11.7 When running: python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-2-7b-chat-hf This solution works for me.

could you check that the problem still exits after rebuilding the repo (pip install -e .)?

[LuJunru]

I met same issue. The problem is I could run vllm on V100 with cuda 11.3, while can not run on A100 with cuda 12.0. I used exact same codes and docker, except cuda.

[Gitwangpin]

I have the same issue. Same nvcc/driver both 11.7 When running: python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-2-7b-chat-hf This solution works for me.

could you check that the problem still exits after rebuilding the repo (pip install -e .)?

I use A100 A40 T4 and use langchain to integrate vllm to encounter this problem in cuda11.7, but I am normal on RTX4090 and RTX3090
Here is the exception output:
INFO 08-25 10:41:25 llm_engine.py:70] Initializing an LLM engine with config: model='meta-llama/Llama-2-7b-chat-hf', tokenizer='meta-llama/Llama-2 -7b-chat-hf', tokenizer_mode=auto, trust_remote_code=False, dtype=torch.float16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=1, seed=0)
INFO 08-25 10:41:25 tokenizer.py:29] For some LLaMA-based models, initializing the fast tokenizer may take a long time. To eliminate the initialization time, consider using 'hf-internal-testing/llama-tokenizer ' instead of the original tokenizer.
Traceback (most recent call last):
File "/root/miniconda3/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/root/miniconda3/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/root/miniconda3/lib/python3.10/site-packages/vllm/entrypoints/api_server.py", line 78, in
engine = AsyncLLMEngine. from_engine_args(engine_args)
File "/root/miniconda3/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 232, in from_engine_args
engine = cls(engine_args. worker_use_ray,
File "/root/miniconda3/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 55, in init
self. engine = engine_class(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 104, in init
self._init_cache()
File "/root/miniconda3/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 182, in _init_cache
num_blocks = self._run_workers(
File "/root/miniconda3/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 470, in _run_workers
output = executor(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/vllm/worker/worker.py", line 108, in profile_num_available_blocks
self.model(
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 253, in forward
hidden_states = self.model(input_ids, positions, kv_caches,
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 220, in forward
hidden_states = layer(
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 172, in forward
hidden_states = self.self_attn(
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 132, in forward
qkv, = self.qkv_proj(hidden_states)
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/vllm/model_executor/parallel_utils/tensor_parallel/layers.py", line 309, in forward
output_parallel = F. linear(input_parallel, self. weight, bias)
RuntimeError: CUDA error: no kernel image is available for execution on the device
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[Gitwangpin]

same the model meta-llama/Llama-2-7b-chat-hf

[lonngxiang]

how to fix？

File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/vllm/model_executor/parallel_utils/tensor_parallel/layers.py", line 309, in forward
output_parallel = F.linear(input_parallel, self.weight, bias)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
root@fa4b127ca2bf:/workspace# python

[chenchenygu]

Also running into this same issue. Has anyone found a fix?

[Symbolk]

Same issue +1! ANY updates or issues? Anyone tried updating Driver Version: 470.141.03 to 515?

[Symbolk]

Same issue +1! ANY updates or issues? Anyone tried updating Driver Version: 470.141.03 to 515?

Solved, by recompiling and reinstalling the lib when deploying on V100. Previously it was compiled on A100.

[YHPeter]

I need to switch between several GPUs (A100, V100, RTX8000) and cuda version, when the CUDA version changes, I need to reinstall it from source. It's a shot-time solution, but it works now!

[shuai-dian]

same issue

root@server-1:~/vllm# python3 -m vllm.entrypoints.api_server --model /workspace/Qwen-7B/weights/Qwen-7B-Chat/ --trust-remote-code
2023-09-25 06:17:03,382 INFO worker.py:1642 -- Started a local Ray instance.
INFO 09-25 06:17:04 llm_engine.py:72] Initializing an LLM engine with config: model='/workspace/Qwen-7B/weights/Qwen-7B-Chat/', tokenizer='/workspace/Qwen-7B/weights/Qwen-7B-Chat/', tokenizer_mode=auto, trust_remote_code=True, dtype=torch.float16, download_dir=None, load_format=auto, tensor_parallel_size=4, seed=0)
WARNING 09-25 06:17:05 tokenizer.py:66] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/root/vllm/vllm/entrypoints/api_server.py", line 177, in
engine = AsyncLLMEngine.from_engine_args(engine_args)
File "/root/vllm/vllm/engine/async_llm_engine.py", line 442, in from_engine_args
engine = cls(engine_args.worker_use_ray,
File "/root/vllm/vllm/engine/async_llm_engine.py", line 250, in init
self.engine = self._init_engine(*args, **kwargs)
File "/root/vllm/vllm/engine/async_llm_engine.py", line 279, in _init_engine
return engine_class(*args, **kwargs)
File "/root/vllm/vllm/engine/llm_engine.py", line 105, in init
self._init_cache()
File "/root/vllm/vllm/engine/llm_engine.py", line 185, in _init_cache
num_blocks = self._run_workers(
File "/root/vllm/vllm/engine/llm_engine.py", line 687, in _run_workers
all_outputs = ray.get(all_outputs)
File "/usr/local/lib/python3.8/dist-packages/ray/_private/auto_init_hook.py", line 24, in auto_init_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/ray/_private/worker.py", line 2547, in get
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(RuntimeError): ray::RayWorker.execute_method() (pid=13558, ip=192.168.1.3, actor_id=60a52a454709a59765dbb35e01000000, repr=<vllm.engine.ray_utils.RayWorker object at 0x7f3ad7c33370>)
File "/root/vllm/vllm/engine/ray_utils.py", line 29, in execute_method
return executor(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/root/vllm/vllm/worker/worker.py", line 108, in profile_num_available_blocks
self.model(
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/vllm/vllm/model_executor/models/qwen.py", line 240, in forward
hidden_states = self.transformer(input_ids, positions, kv_caches,
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/vllm/vllm/model_executor/models/qwen.py", line 205, in forward
hidden_states = layer(
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/vllm/vllm/model_executor/models/qwen.py", line 171, in forward
hidden_states = residual + hidden_states
RuntimeError: CUDA error: no kernel image is available for execution on the device

I use cuda118 docker，builded in A800 , The Same Docker copy to H100 GPU, When it restart, there was this issue, The problem persists after rebuild

[schnurro-bits]

I've been having the same issue for the last week.

GPU: Titan XP with CUDA 12.0
nvcc -V == 11.8
torch.version.cuda == 11.8

I've tried reinstalling PyTorch, which did not resolve the problem.

[Fr4nk1inCs]

I've been having the same issue for the last week.

GPU: Titan XP with CUDA 12.0 nvcc -V == 11.8 torch.version.cuda == 11.8

I've tried reinstalling PyTorch, which did not resolve the problem.

@schnurromafia I don't think Titan Xp is supported, since its compute capacity is 6.1. One of vLLM's requirements is GPU with compute capability 7.0 or higher.