[Bug]: meta-llama/Llama-4-Scout-17B-16E-Instruct compatibility

[alokkrsahu]

Your current environment

Error while deploying the LLM

🐛 Describe the bug

Loading safetensors checkpoint shards: 96% Completed | 48/50 [00:03<00:00, 25.21it/s]
Loading safetensors checkpoint shards: 100% Completed | 50/50 [00:03<00:00, 13.83it/s]
(VllmWorker rank=0 pid=2770071)
(VllmWorker rank=0 pid=2770071) INFO 04-08 16:26:46 [loader.py:447] Loading weights took 177.20 seconds
(VllmWorker rank=0 pid=2770071) INFO 04-08 16:26:47 [gpu_model_runner.py:1273] Model loading took 53.1198 GiB and 178.255213 seconds
(VllmWorker rank=1 pid=2770083) INFO 04-08 16:26:49 [loader.py:447] Loading weights took 180.12 seconds
(VllmWorker rank=2 pid=2770173) INFO 04-08 16:26:49 [loader.py:447] Loading weights took 180.84 seconds
(VllmWorker rank=1 pid=2770083) INFO 04-08 16:26:49 [gpu_model_runner.py:1273] Model loading took 53.1198 GiB and 181.444976 seconds
(VllmWorker rank=2 pid=2770173) INFO 04-08 16:26:49 [gpu_model_runner.py:1273] Model loading took 53.1198 GiB and 182.203734 seconds
(VllmWorker rank=3 pid=2770198) INFO 04-08 16:26:49 [loader.py:447] Loading weights took 181.73 seconds
(VllmWorker rank=3 pid=2770198) INFO 04-08 16:26:50 [gpu_model_runner.py:1273] Model loading took

what(): CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f8f1ef6c1b6 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f8f1ef15a76 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f8f1f355918 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #3: + 0x103ad78 (0x7f8ecd065d78 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x10433c5 (0x7f8ecd06e3c5 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: + 0x6417b2 (0x7f8f168d07b2 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #6: + 0x6f30f (0x7f8f1ef4d30f in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x21b (0x7f8f1ef4633b in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f8f1ef464e9 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #9: + 0x8fefb8 (0x7f8f16b8dfb8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #10: THPVariable_subclass_dealloc(_object*) + 0x2f6 (0x7f8f16b8e306 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #11: + 0x13c035 (0x7f8e5f4a5035 in /usr/local/lib/python3.10/dist-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so)

what(): CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f8f1ef6c1b6 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f8f1ef15a76 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f8f1f355918 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #3: + 0x103ad78 (0x7f8ecd065d78 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x10433c5 (0x7f8ecd06e3c5 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: + 0x6417b2 (0x7f8f168d07b2 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #6: + 0x6f30f (0x7f8f1ef4d30f in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x21b (0x7f8f1ef4633b in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f8f1ef464e9 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #9: + 0x8fefb8 (0x7f8f16b8dfb8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #10: THPVariable_subclass_dealloc(_object*) + 0x2f6 (0x7f8f16b8e306 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #11: + 0x13c035 (0x7f8e5f4a5035 in /usr/local/lib/python3.10/dist-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so)

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[sarckk]

@alokkrsahu could you share the command you are running? also try rebasing after #16209 if you are using max-model-len < 8192.

[alokkrsahu]

Hi, I was able to solve that error using the below command. PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True TORCH_COMPILE_CACHE_DIR=/data/llama4/torch_cache XDG_CACHE_HOME=/data/llama4/torch_cache vllm serve "meta-llama/Llama-4-Scout-17B-16E-Instruct" --tensor-parallel-size 4 --gpu-memory-utilization 0.95 --max-model-len 49152 --block-size 16 --swap-space 16 --enforce-eager --max-num-batched-tokens 8192 --max-num-seqs 256 Please let me know if this is the best way to deploy it on 4xH100 Sent from Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Yong Hoon Shin ***@***.***> Sent: Wednesday, April 9, 2025 5:31:35 PM To: vllm-project/vllm ***@***.***> Cc: Alok Kumar Sahu ***@***.***>; Mention ***@***.***> Subject: Re: [vllm-project/vllm] [Bug]: meta-llama/Llama-4-Scout-17B-16E-Instruct compatibility (Issue #16330) @alokkrsahu<https://github.com/alokkrsahu> could you share the command you are running? also try rebasing after #16209<#16209> if you are using max-model-len < 8192. — Reply to this email directly, view it on GitHub<#16330 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AMDHPKHG6TQXQT7RD6MAQ6T2YVDOPAVCNFSM6AAAAAB2YO5HHOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJQGMZTEMZSGU>. You are receiving this because you were mentioned.Message ID: ***@***.***> [https://avatars.githubusercontent.com/u/48474650?s=20&v=4]sarckk left a comment (vllm-project/vllm#16330)<#16330 (comment)> @alokkrsahu<https://github.com/alokkrsahu> could you share the command you are running? also try rebasing after #16209<#16209> if you are using max-model-len < 8192. — Reply to this email directly, view it on GitHub<#16330 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AMDHPKHG6TQXQT7RD6MAQ6T2YVDOPAVCNFSM6AAAAAB2YO5HHOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJQGMZTEMZSGU>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[sarckk]

That seems fine, I think you were likely using --max-model-len that was less than 8192 (local attention chunk size). That issue has since been resolved, so you can now specify max model len < 8192.

Closing as you were able to resolve it, but let me know if you run into other issues.