Revert "[CI] Update FlashInfer to 0.2.6.post1" --- edit: No, better add "12.0" to FlashInfer TORCH_CUDA_ARCH_LIST see PR #19794

[cyril23]

The Pullrequest #19297 broke SM 120 Blackwell compability (RTX 50xx, RTX PRO).

You can't use -e VLLM_USE_FLASHINFER_SAMPLER=1 anymore (which is the default) and need to fall back to -e VLLM_USE_FLASHINFER_SAMPLER=0 which will give you less performance and this warning:

WARNING 06-18 08:55:01 [topk_topp_sampler.py:52] FlashInfer is available, but it is not enabled. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please set VLLM_USE_FLASHINFER_SAMPLER=1.

I did 2 builds, both with --build-arg torch_cuda_arch_list='12.0 (SM 120 compatible only) and pushed them to Docker hub:

1. wurstdeploy/vllm:azure10thjunesolo120 which is based on the last commit of 10th June (da9b523) and which still uses the old FlashInfer version

git checkout -b 10thjune da9b523ce1fd5c27bfd18921ba0388bf2e8e4618
DOCKER_BUILDKIT=1 sudo docker build --build-arg max_jobs=64   --build-arg USE_SCCACHE=0 --build-arg GIT_REPO_CHECK=1   --build-arg CUDA_VERSION=12.8.1   --build-arg torch_cuda_arch_list='12.0'   --build-arg RUN_WHEEL_CHECK=false   --tag wurstdeploy/vllm:azure10thjunesolo120 --target vllm-openai   --progress plain -f docker/Dockerfile .

# this is still SM 120 compatible, you can run via
sudo docker run --runtime nvidia --gpus all     -v ~/.cache/huggingface:/root/.cache/huggingface     -p 8000:8000 \
  -e VLLM_USE_FLASHINFER_SAMPLER=1 \
  wurstdeploy/vllm:azure10thjunesolo120    --model Qwen/Qwen3-0.6B

2. wurstdeploy/vllm:azure11thjunesolo120 which is based on the last commit of 11th June (42f52cc) and already includes the commit 497a91e and therefore the updated Flashinfer version

git checkout -b 11thjune 42f52cc95bf34a2e15f4cdbc8474503a9bcc970f
DOCKER_BUILDKIT=1 sudo docker build --build-arg max_jobs=64   --build-arg USE_SCCACHE=0 --build-arg GIT_REPO_CHECK=1   --build-arg CUDA_VERSION=12.8.1   --build-arg torch_cuda_arch_list='12.0'   --build-arg RUN_WHEEL_CHECK=false   --tag wurstdeploy/vllm:azure11thjunesolo120 --target vllm-openai   --progress plain -f docker/Dockerfile .

# this is not fully SM 120 compatible anymore:
sudo docker run --runtime nvidia --gpus all     -v ~/.cache/huggingface:/root/.cache/huggingface     -p 8000:8000 \
  -e VLLM_USE_FLASHINFER_SAMPLER=1 \
  wurstdeploy/vllm:azure11thjunesolo120    --model Qwen/Qwen3-0.6B

INFO 06-18 08:53:41 [monitor.py:34] torch.compile takes 18.01 s in total
/usr/local/lib/python3.12/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
Process EngineCore_0:
ERROR 06-18 08:53:41 [core.py:515] EngineCore failed to start.
ERROR 06-18 08:53:41 [core.py:515] Traceback (most recent call last):
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 506, in run_engine_core
ERROR 06-18 08:53:41 [core.py:515]     engine_core = EngineCoreProc(*args, **kwargs)
ERROR 06-18 08:53:41 [core.py:515]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 390, in __init__
ERROR 06-18 08:53:41 [core.py:515]     super().__init__(vllm_config, executor_class, log_stats,
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 83, in __init__
ERROR 06-18 08:53:41 [core.py:515]     self._initialize_kv_caches(vllm_config)
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 141, in _initialize_kv_caches
ERROR 06-18 08:53:41 [core.py:515]     available_gpu_memory = self.model_executor.determine_available_memory()
ERROR 06-18 08:53:41 [core.py:515]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 76, in determine_available_memory
ERROR 06-18 08:53:41 [core.py:515]     output = self.collective_rpc("determine_available_memory")
ERROR 06-18 08:53:41 [core.py:515]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 57, in collective_rpc
ERROR 06-18 08:53:41 [core.py:515]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 06-18 08:53:41 [core.py:515]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2680, in run_method
ERROR 06-18 08:53:41 [core.py:515]     return func(*args, **kwargs)
ERROR 06-18 08:53:41 [core.py:515]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 06-18 08:53:41 [core.py:515]     return func(*args, **kwargs)
ERROR 06-18 08:53:41 [core.py:515]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 205, in determine_available_memory
ERROR 06-18 08:53:41 [core.py:515]     self.model_runner.profile_run()
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2015, in profile_run
ERROR 06-18 08:53:41 [core.py:515]     sampler_output = self._dummy_sampler_run(hidden_states)
ERROR 06-18 08:53:41 [core.py:515]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 06-18 08:53:41 [core.py:515]     return func(*args, **kwargs)
ERROR 06-18 08:53:41 [core.py:515]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1913, in _dummy_sampler_run
ERROR 06-18 08:53:41 [core.py:515]     raise e
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1903, in _dummy_sampler_run
ERROR 06-18 08:53:41 [core.py:515]     sampler_output = self.sampler(logits=logits,
ERROR 06-18 08:53:41 [core.py:515]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
ERROR 06-18 08:53:41 [core.py:515]     return self._call_impl(*args, **kwargs)
ERROR 06-18 08:53:41 [core.py:515]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
ERROR 06-18 08:53:41 [core.py:515]     return forward_call(*args, **kwargs)
ERROR 06-18 08:53:41 [core.py:515]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/sample/sampler.py", line 52, in forward
ERROR 06-18 08:53:41 [core.py:515]     sampled = self.sample(logits, sampling_metadata)
ERROR 06-18 08:53:41 [core.py:515]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/sample/sampler.py", line 118, in sample
ERROR 06-18 08:53:41 [core.py:515]     random_sampled = self.topk_topp_sampler(
ERROR 06-18 08:53:41 [core.py:515]                      ^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
ERROR 06-18 08:53:41 [core.py:515]     return self._call_impl(*args, **kwargs)
ERROR 06-18 08:53:41 [core.py:515]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
ERROR 06-18 08:53:41 [core.py:515]     return forward_call(*args, **kwargs)
ERROR 06-18 08:53:41 [core.py:515]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/sample/ops/topk_topp_sampler.py", line 104, in forward_cuda
ERROR 06-18 08:53:41 [core.py:515]     return flashinfer_sample(logits, k, p, generators)
ERROR 06-18 08:53:41 [core.py:515]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/sample/ops/topk_topp_sampler.py", line 290, in flashinfer_sample
ERROR 06-18 08:53:41 [core.py:515]     next_token_ids = flashinfer.sampling.top_k_top_p_sampling_from_logits(
ERROR 06-18 08:53:41 [core.py:515]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/flashinfer/sampling.py", line 901, in top_k_top_p_sampling_from_logits
ERROR 06-18 08:53:41 [core.py:515]     masked_logits = top_k_mask_logits(logits, top_k)
ERROR 06-18 08:53:41 [core.py:515]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/flashinfer/sampling.py", line 1221, in top_k_mask_logits
ERROR 06-18 08:53:41 [core.py:515]     return get_sampling_module().top_k_mask_logits(
ERROR 06-18 08:53:41 [core.py:515]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/flashinfer/sampling.py", line 352, in top_k_mask_logits
ERROR 06-18 08:53:41 [core.py:515]     module.top_k_mask_logits.default(
ERROR 06-18 08:53:41 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 756, in __call__
ERROR 06-18 08:53:41 [core.py:515]     return self._op(*args, **kwargs)
ERROR 06-18 08:53:41 [core.py:515]            ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-18 08:53:41 [core.py:515] RuntimeError: TopKMaskLogits failed with error code no kernel image is available for execution on the device


# you can only run it without Flashinfer, i.e. -e VLLM_USE_FLASHINFER_SAMPLER=0:
sudo docker run --runtime nvidia --gpus all     -v ~/.cache/huggingface:/root/.cache/huggingface     -p 8000:8000 \
  -e VLLM_USE_FLASHINFER_SAMPLER=0 \
  wurstdeploy/vllm:azure11thjunesolo120    --model Qwen/Qwen3-0.6B
> WARNING 06-18 08:55:01 [topk_topp_sampler.py:52] FlashInfer is available, but it is not enabled. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please set VLLM_USE_FLASHINFER_SAMPLER=1.

Originally posted by @cyril23 in #19297 (comment)

[cyril23]

Or maybe just add 12.0 to TORCH_CUDA_ARCH_LIST in order to make it SM120 compatible, see https://github.com/vllm-project/vllm/pull/19794/files#diff-f34da55ca08f1a30591d8b0b3e885bcc678537b2a9a4aadea4f190806b374ddcR264

That would probably be better

[cyril23]

I've made another build:

3. based on the current https://github.com/vllm-project/vllm main branch (Commit 8b6e1d6 from 18th June), with the commit 497a91e from 11th June reverted as follows:
   file revert.patch

diff --git a/docker/Dockerfile b/docker/Dockerfile
index cf9c245a9..c41c6ef0c 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -259,16 +259,14 @@ RUN --mount=type=cache,target=/root/.cache/uv \
 if [ "$TARGETPLATFORM" != "linux/arm64" ]; then \
     # FlashInfer already has a wheel for PyTorch 2.7.0 and CUDA 12.8. This is enough for CI use
     if [[ "$CUDA_VERSION" == 12.8* ]]; then \
-        uv pip install --system https://download.pytorch.org/whl/cu128/flashinfer/flashinfer_python-0.2.6.post1%2Bcu128torch2.7-cp39-abi3-linux_x86_64.whl; \
+        uv pip install --system https://download.pytorch.org/whl/cu128/flashinfer/flashinfer_python-0.2.5%2Bcu128torch2.7-cp38-abi3-linux_x86_64.whl; \
     else \
-        export TORCH_CUDA_ARCH_LIST='7.5 8.0 8.9 9.0a 10.0a' && \
-        git clone https://github.com/flashinfer-ai/flashinfer.git --single-branch --branch v0.2.6.post1 --recursive && \
-        # Needed to build AOT kernels
-        (cd flashinfer && \
-            python3 -m flashinfer.aot && \
-            uv pip install --system --no-build-isolation . \
-        ) && \
-        rm -rf flashinfer; \
+        export TORCH_CUDA_ARCH_LIST='7.5 8.0 8.9 9.0+PTX'; \
+        CUDA_MAJOR="${CUDA_VERSION%%.*}"; \
+        if [ "$CUDA_MAJOR" -lt 12 ]; then \
+            export FLASHINFER_ENABLE_SM90=0; \
+        fi; \
+        uv pip install --system --no-build-isolation "git+https://github.com/flashinfer-ai/flashinfer@21ea1d2545f74782b91eb8c08fd503ac4c0743fc" ; \
     fi \
 fi
 COPY examples examples

Build via

DOCKER_BUILDKIT=1 sudo docker build \
  --build-arg max_jobs=5 \
  --build-arg USE_SCCACHE=0 \
  --build-arg GIT_REPO_CHECK=0 \
  --build-arg CUDA_VERSION=12.8.1 \
  --build-arg torch_cuda_arch_list='12.0' \
  --build-arg RUN_WHEEL_CHECK=false \
  --tag wurstdeploy/vllm:revertedpr19297 \
  --target vllm-openai \
  --progress plain \
  -f docker/Dockerfile .

Run via

sudo docker run --runtime nvidia --gpus all     -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  -e VLLM_USE_FLASHINFER_SAMPLER=1 \
  wurstdeploy/vllm:revertedpr19297
  --model Qwen/Qwen3-0.6B

It works:

INFO 06-18 15:48:37 [__init__.py:244] Automatically detected platform cuda.
INFO 06-18 15:48:39 [api_server.py:1287] vLLM API server version 0.9.2.dev140+g8b6e1d639.d20250618
INFO 06-18 15:48:40 [cli_args.py:309] non-default args: {'model': 'Qwen/Qwen3-0.6B'}
INFO 06-18 15:48:44 [config.py:831] This model supports multiple tasks: {'score', 'classify', 'reward', 'generate', 'embed'}. Defaulting to 'generate'.
INFO 06-18 15:48:44 [config.py:1444] Using max model len 40960
INFO 06-18 15:48:44 [config.py:2197] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 06-18 15:48:47 [__init__.py:244] Automatically detected platform cuda.
INFO 06-18 15:48:48 [core.py:460] Waiting for init message from front-end.
INFO 06-18 15:48:48 [core.py:70] Initializing a V1 LLM engine (v0.9.2.dev140+g8b6e1d639.d20250618) with config: model='Qwen/Qwen3-0.6B', speculative_config=None, tokenizer='Qwen/Qwen3-0.6B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen/Qwen3-0.6B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null}
WARNING 06-18 15:48:49 [utils.py:2756] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7fe3bc23bbc0>
INFO 06-18 15:48:49 [parallel_state.py:1072] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
WARNING 06-18 15:48:49 [interface.py:376] Using 'pin_memory=False' as WSL is detected. This may slow down the performance.
INFO 06-18 15:48:49 [topk_topp_sampler.py:49] Using FlashInfer for top-p & top-k sampling.
INFO 06-18 15:48:49 [gpu_model_runner.py:1627] Starting to load model Qwen/Qwen3-0.6B...
INFO 06-18 15:48:49 [gpu_model_runner.py:1632] Loading model from scratch...
INFO 06-18 15:48:49 [cuda.py:259] Using Flash Attention backend on V1 engine.
INFO 06-18 15:48:50 [weight_utils.py:292] Using model weights format ['*.safetensors']
INFO 06-18 15:48:50 [weight_utils.py:345] No model.safetensors.index.json found in remote.
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  8.03it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  8.02it/s]

INFO 06-18 15:48:50 [default_loader.py:272] Loading weights took 0.15 seconds
INFO 06-18 15:48:51 [gpu_model_runner.py:1656] Model loading took 1.1201 GiB and 0.978284 seconds
INFO 06-18 15:48:54 [backends.py:508] Using cache directory: /root/.cache/vllm/torch_compile_cache/7b7604ccca/rank_0_0/backbone for vLLM's torch.compile
INFO 06-18 15:48:54 [backends.py:519] Dynamo bytecode transform time: 3.46 s
INFO 06-18 15:48:56 [backends.py:181] Cache the graph of shape None for later use
INFO 06-18 15:49:09 [backends.py:193] Compiling a graph for general shape takes 15.05 s
INFO 06-18 15:49:19 [monitor.py:34] torch.compile takes 18.51 s in total
INFO 06-18 15:49:27 [gpu_worker.py:232] Available KV cache memory: 26.91 GiB
INFO 06-18 15:49:27 [kv_cache_utils.py:716] GPU KV cache size: 251,952 tokens
INFO 06-18 15:49:27 [kv_cache_utils.py:720] Maximum concurrency for 40,960 tokens per request: 6.15x
WARNING 06-18 15:49:27 [utils.py:101] Unable to detect current VLLM config. Defaulting to NHD kv cache layout.
Capturing CUDA graphs: 100%|██████████| 67/67 [00:13<00:00,  5.08it/s]
INFO 06-18 15:49:40 [gpu_model_runner.py:2083] Graph capturing finished in 13 secs, took 0.75 GiB
INFO 06-18 15:49:40 [core.py:173] init engine (profile, create kv cache, warmup model) took 49.92 seconds
INFO 06-18 15:49:41 [loggers.py:137] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 15747
WARNING 06-18 15:49:41 [config.py:1371] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
INFO 06-18 15:49:41 [serving_chat.py:118] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
INFO 06-18 15:49:41 [serving_completion.py:66] Using default completion sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
INFO 06-18 15:49:41 [api_server.py:1349] Starting vLLM API server 0 on http://0.0.0.0:8000
INFO 06-18 15:49:41 [launcher.py:29] Available routes are:
INFO 06-18 15:49:41 [launcher.py:37] Route: /openapi.json, Methods: GET, HEAD
INFO 06-18 15:49:41 [launcher.py:37] Route: /docs, Methods: GET, HEAD
INFO 06-18 15:49:41 [launcher.py:37] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 06-18 15:49:41 [launcher.py:37] Route: /redoc, Methods: GET, HEAD
INFO 06-18 15:49:41 [launcher.py:37] Route: /health, Methods: GET
INFO 06-18 15:49:41 [launcher.py:37] Route: /load, Methods: GET
INFO 06-18 15:49:41 [launcher.py:37] Route: /ping, Methods: POST
INFO 06-18 15:49:41 [launcher.py:37] Route: /ping, Methods: GET
INFO 06-18 15:49:41 [launcher.py:37] Route: /tokenize, Methods: POST
INFO 06-18 15:49:41 [launcher.py:37] Route: /detokenize, Methods: POST
INFO 06-18 15:49:41 [launcher.py:37] Route: /v1/models, Methods: GET
INFO 06-18 15:49:41 [launcher.py:37] Route: /version, Methods: GET
INFO 06-18 15:49:41 [launcher.py:37] Route: /v1/chat/completions, Methods: POST
INFO 06-18 15:49:41 [launcher.py:37] Route: /v1/completions, Methods: POST
INFO 06-18 15:49:41 [launcher.py:37] Route: /v1/embeddings, Methods: POST
INFO 06-18 15:49:41 [launcher.py:37] Route: /pooling, Methods: POST
INFO 06-18 15:49:41 [launcher.py:37] Route: /classify, Methods: POST
INFO 06-18 15:49:41 [launcher.py:37] Route: /score, Methods: POST
INFO 06-18 15:49:41 [launcher.py:37] Route: /v1/score, Methods: POST
INFO 06-18 15:49:41 [launcher.py:37] Route: /v1/audio/transcriptions, Methods: POST
INFO 06-18 15:49:41 [launcher.py:37] Route: /rerank, Methods: POST
INFO 06-18 15:49:41 [launcher.py:37] Route: /v1/rerank, Methods: POST
INFO 06-18 15:49:41 [launcher.py:37] Route: /v2/rerank, Methods: POST
INFO 06-18 15:49:41 [launcher.py:37] Route: /invocations, Methods: POST
INFO 06-18 15:49:41 [launcher.py:37] Route: /metrics, Methods: GET
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO 06-18 15:49:44 [chat_utils.py:420] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
INFO 06-18 15:49:44 [logger.py:43] Received request chatcmpl-0677acceae694c31b9231d24f141404c: prompt: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n/no_think What is the capital of France? Tell me 2 sentences about it<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=0.95, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=40924, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None.
INFO 06-18 15:49:44 [async_llm.py:271] Added request chatcmpl-0677acceae694c31b9231d24f141404c.

and I've deployed this image wurstdeploy/vllm:revertedpr19297 to Docker hub sinceit is a pretty up to date SM120 capable image.

The better solution is still:

Or maybe just add 12.0 to TORCH_CUDA_ARCH_LIST in order to make it SM120 compatible, see https://github.com/vllm-project/vllm/pull/19794/files#diff-f34da55ca08f1a30591d8b0b3e885bcc678537b2a9a4aadea4f190806b374ddcR264

That would probably be better

So I'll close this issue here in favor of PR #19794 which has been already approved and includes a better solution.