Closed
Description
Your current environment
I've tried nightly builds, building from source, and docker containers, but I am always seeing the same error:
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
I am able to reproduce the issue by running the following container on Ubuntu 24.04:
docker run --rm \
--gpus all \
-v "${HOME}/.cache/huggingface:/root/.cache/huggingface" \
-p 8000:8000 \
-e VLLM_USE_FLASH_ATTN=0 \
--ipc=host \
vllm/vllm-openai:latest \
--model "${MODEL_NAME}"
Where ${MODEL_NAME} for example could be Qwen/Qwen3-4B-AWQ.
nvidia-smi output:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20 Driver Version: 570.133.20 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 5090 Off | 00000000:01:00.0 Off | N/A |
| 0% 35C P8 12W / 575W | 15MiB / 32607MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 1553 G /usr/lib/xorg/Xorg 4MiB |
+-----------------------------------------------------------------------------------------+
🐛 Describe the bug
I am able to reproduce the bug from the following simple example:
curl -X POST http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "Qwen/Qwen3-4B-AWQ",
"messages": [
{ "role": "system", "content": "You are a helpful assistant." },
{ "role": "user", "content": "Tell me a joke." }
],
"max_tokens": 50,
"temperature": 0.7
}'
FYI ollama works fine as a backend with the 5090, but I want the improved performance and tool calling ability which comes with vllm (OpenAI compatibility). If needed, I can provide the full debug output. I've tried with and without the VLLM_USE_FLASH_ATTN environment variable.
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.