cached tokens completions #22

tanujtiwari1998 · 2025-07-08T17:08:49Z

[Benchmarks] Remove invalid option under V1 engine ([Benchmarks] Remove invalid option under V1 engine vllm-project/vllm#17651)
[BugFix] Increase timeout for startup failure test ([BugFix] Increase timeout for startup failure test vllm-project/vllm#17642)
[TPU] Enable gemma3-27b with TP>1 on multi-chips. ([TPU] Enable gemma3-27b with TP>1 on multi-chips. vllm-project/vllm#17335)
[TPU][V1] Add support for top-logprobs ([TPU][V1] Add support for top-logprobs vllm-project/vllm#17072)
[Bugfix] LoRA - Retire unused maxnreg LoRA kernel argument ([Bugfix] LoRA - Retire unused maxnreg LoRA kernel argument vllm-project/vllm#17677)
Update nm to rht in doc links + refine fp8 doc (Update nm to rht in doc links + refine fp8 doc vllm-project/vllm#17678)
[Model] Add GraniteMoeHybrid 4.0 model ([Model] Add GraniteMoeHybrid 4.0 model vllm-project/vllm#17497)
[easy] Fix logspam on PiecewiseBackend errors ([easy] Fix logspam on PiecewiseBackend errors vllm-project/vllm#17138)
[Bugfix] Fixed prompt length for random dataset ([Bugfix] Fixed prompt length for random dataset vllm-project/vllm#17408)
[Doc] Update notes for H2O-VL and Gemma3 ([Doc] Update notes for H2O-VL and Gemma3 vllm-project/vllm#17219)
[Misc] Fix ScalarType float4 naming ([Misc] Fix ScalarType float4 naming vllm-project/vllm#17690)
Fix dockerfilegraph pre-commit hook (Fix dockerfilegraph pre-commit hook vllm-project/vllm#17698)
[Bugfix] Fix triton import with local TritonPlaceholder ([Bugfix] Fix triton import with local TritonPlaceholder vllm-project/vllm#17446)
[V1] Enable TPU V1 backend by default ([V1] Enable TPU V1 backend by default vllm-project/vllm#17673)
[V1][PP] Support PP for MultiprocExecutor ([V1][PP] Support PP for MultiprocExecutor vllm-project/vllm#14219)
[v1] AttentionMetadata for each layer ([v1] AttentionMetadata for each layer vllm-project/vllm#17394)
[Feat] Add deprecated=True to CLI args ([Feat] Add deprecated=True to CLI args vllm-project/vllm#17426)
[Docs] Use gh-file to add links to tool_calling.md ([Docs] Use gh-file to add links to tool_calling.md vllm-project/vllm#17709)
[v1] Introduce KVCacheBlocks as interface between Scheduler and KVCacheManager ([v1] Introduce KVCacheBlocks as interface between Scheduler and KVCacheManager vllm-project/vllm#17479)
[doc] Add RAG Integration example ([doc] Add RAG Integration example vllm-project/vllm#17692)
[Bugfix] Fix modality limits in vision language example ([Bugfix] Fix modality limits in vision language example vllm-project/vllm#17721)
Make right sidebar more readable in "Supported Models" (Make right sidebar more readable in "Supported Models" vllm-project/vllm#17723)
[TPU] Increase block size and reset block shapes ([TPU] Increase block size and reset block shapes vllm-project/vllm#16458)
[Misc] Add Next Edit Prediction (NEP) datasets support in benchmark_serving.py ([Misc] Add Next Edit Prediction (NEP) datasets support in benchmark_serving.py vllm-project/vllm#16839)
[Bugfix] Fix for the condition to accept empty encoder inputs for mllama ([Bugfix] Fix for the condition to accept empty encoder inputs for mllama vllm-project/vllm#17732)
[Kernel] Unified Triton kernel that doesn't distinguish between prefill + decode ([Kernel] Unified Triton kernel that doesn't distinguish between prefill + decode vllm-project/vllm#16828)
Fix doc build performance (Fix doc build performance vllm-project/vllm#17748)
[ROCm] fix num_stages for default moe config to avoid triton OutOfResource error ([ROCm] fix num_stages for default moe config to avoid triton OutOfResource error vllm-project/vllm#17744)
Add logging for torch nightly version (Add logging for torch nightly version vllm-project/vllm#17669)
[Model] Mamba2 causal conv1d Refactor to Split Prefill and Decode Requests for Corresponding Kernels ([Model] Mamba2 causal conv1d Refactor to Split Prefill and Decode Requests for Corresponding Kernels vllm-project/vllm#17146)
Removed unused marlin cuda code (Removed unused marlin cuda code vllm-project/vllm#17684)
[TPU] Add kernel test for moe_pallas ([TPU] Add kernel test for moe_pallas vllm-project/vllm#17496)
Replace lm-eval bash script with pytest and use enforce_eager for faster CI (Replace lm-eval bash script with pytest and use enforce_eager for faster CI vllm-project/vllm#17717)
[BugFix][Spec Decode] Fix hidden size mismatch between target and eagle head ([BugFix][Spec Decode] Fix hidden size mismatch between target and eagle head vllm-project/vllm#17740)
[Misc] Split model loader ([Misc] Split model loader vllm-project/vllm#17712)
[Misc] Use apply_rotary_emb from vllm_flash_attn for Qwen2-VL vision RoPE ([Misc] Use apply_rotary_emb from vllm_flash_attn for Qwen2-VL vision RoPE vllm-project/vllm#17726)
[Kernel] GGUF MoeVec kernel ([Kernel] GGUF MoeVec kernel vllm-project/vllm#16780)
[Kernel] Use fused rmsnorm for some models like qwen3 series ([Kernel] Use fused rmsnorm for some models like qwen3 series vllm-project/vllm#17735)
[Misc] Remove qlora_adapter_name_or_path ([Misc] Remove qlora_adapter_name_or_path vllm-project/vllm#17699)
Add NeuronxDistributedInference support, Speculative Decoding, Dynamic on-device sampling (Add NeuronxDistributedInference support, Speculative Decoding, Dynamic on-device sampling vllm-project/vllm#16357)
[Frontend] Add missing chat templates for various MLLMs ([Frontend] Add missing chat templates for various MLLMs vllm-project/vllm#17758)
Fix test_memory_usage_no_spec (Fix test_memory_usage_no_spec vllm-project/vllm#17754)
Make key optional for rotary embedding (Make key optional for rotary embedding vllm-project/vllm#17566)
[doc] update the issue link ([doc] update the issue link vllm-project/vllm#17782)
[ROCm][FP8][Kernel] FP8 quantization fused into Custom Paged Attention ([ROCm][FP8][Kernel] FP8 quantization fused into Custom Paged Attention vllm-project/vllm#17139)
Only depend on importlib-metadata for Python < 3.10 (Only depend on importlib-metadata for Python < 3.10 vllm-project/vllm#17776)
[Bugfix] Fix Video IO error for short video ([Bugfix] Fix Video IO error for short video vllm-project/vllm#17791)
Fix and simplify deprecated=True CLI kwarg (Fix and simplify deprecated=True CLI kwarg vllm-project/vllm#17781)
[Bugfix] Fix missing lora name mapping for lora without prefix ([Bugfix] Fix missing lora name mapping for lora without prefix vllm-project/vllm#17793)
[Quantization] Quark MXFP4 format loading ([Quantization] Quark MXFP4 format loading vllm-project/vllm#16943)
[Hardware][TPU][V1] Multi-LoRA implementation for the V1 TPU backend ([Hardware][TPU][V1] Multi-LoRA implementation for the V1 TPU backend vllm-project/vllm#14238)
[BugFix] Avoid secondary missing MultiprocExecutor.workers error ([BugFix] Avoid secondary missing MultiprocExecutor.workers error vllm-project/vllm#17811)
[Core][Feature] Input metadata dump on crash ([Core][Feature] Input metadata dump on crash vllm-project/vllm#13407)
[Chore][Doc] uses model id determined from OpenAI client ([Chore][Doc] uses model id determined from OpenAI client vllm-project/vllm#17815)
Don't call the venv vllm (Don't call the venv vllm vllm-project/vllm#17810)
[BugFix] Fix --disable-log-stats in V1 server mode ([BugFix] Fix --disable-log-stats in V1 server mode vllm-project/vllm#17600)
[Core] Support full cuda graph in v1 ([Core] Support full cuda graph in v1 vllm-project/vllm#16072)
Improve exception reporting in MP engine (Improve exception reporting in MP engine vllm-project/vllm#17800)
[Installation] OpenTelemetry version update ([Installation] OpenTelemetry version update vllm-project/vllm#17771)
Only log non-default CLI args for online serving (Only log non-default CLI args for online serving vllm-project/vllm#17803)
[V1] Add VLLM_ALLOW_INSECURE_SERIALIZATION env var ([V1] Add VLLM_ALLOW_INSECURE_SERIALIZATION env var vllm-project/vllm#17490)
[Kernel][Hardware][AMD] Bf16 mfma opt for ROCm skinny GEMMs ([Kernel][Hardware][AMD] Bf16 mfma opt for ROCm skinny GEMMs vllm-project/vllm#17071)
[Hardware][Power] Enable compressed tensor W8A8 INT8 quantization for POWER ([Hardware][Power] Enable compressed tensor W8A8 INT8 quantization for POWER vllm-project/vllm#17153)
[Hardware][Intel-Gaudi] Support Automatic Prefix Caching on HPU ([Hardware][Intel-Gaudi] Support Automatic Prefix Caching on HPU vllm-project/vllm#17648)
[Frontend] Chat template fallbacks for multimodal models ([Frontend] Chat template fallbacks for multimodal models vllm-project/vllm#17805)
[Qwen3]add qwen3-235b-bf16 fused moe config on A100 ([Qwen3]add qwen3-235b-bf16 fused moe config on A100 vllm-project/vllm#17715)
[Bugfix] Fix bad words for Mistral models ([Bugfix] Fix bad words for Mistral models vllm-project/vllm#17753)
[Misc] support model prefix & add deepseek vl2 tiny fused moe config ([Misc] support model prefix & add deepseek vl2 tiny fused moe config vllm-project/vllm#17763)
[Bugfix] Fix tool call template validation for Mistral models ([Bugfix] Fix tool call template validation for Mistral models vllm-project/vllm#17644)
[TPU] Fix the test_sampler ([TPU] Fix the test_sampler vllm-project/vllm#17820)
[Bugfix] Fix quark fp8 format loading on AMD GPUs ([Bugfix] Fix quark fp8 format loading on AMD GPUs vllm-project/vllm#12612)
[Doc] Fix a typo in the file name ([Doc] Fix a typo in the file name vllm-project/vllm#17836)
[Easy] Eliminate c10::optional usage in vllm/csrc ([Easy] Eliminate c10::optional usage in vllm/csrc vllm-project/vllm#17819)
[Misc] add chatbox integration ([Misc] add chatbox integration vllm-project/vllm#17828)
Fix transient dependency error in docs build (Fix transient dependency error in docs build vllm-project/vllm#17848)
[Bugfix] use_fast failing to be propagated to Qwen2-VL image processor ([Bugfix] use_fast failing to be propagated to Qwen2-VL image processor vllm-project/vllm#17838)
[Misc] Delete LoRA-related redundancy code ([Misc] Delete LoRA-related redundancy code vllm-project/vllm#17841)
[CI] Fix test_collective_rpc ([CI] Fix test_collective_rpc vllm-project/vllm#17858)
[V1] Improve VLLM_ALLOW_INSECURE_SERIALIZATION logging ([V1] Improve VLLM_ALLOW_INSECURE_SERIALIZATION logging vllm-project/vllm#17860)
[Test] Attempt all TPU V1 tests, even if some of them fail. ([Test] Attempt all TPU V1 tests, even if some of them fail. vllm-project/vllm#17334)
[CI] Prune down lm-eval small tests ([CI] Prune down lm-eval small tests vllm-project/vllm#17012)
Fix noisy warning for uncalibrated q_scale/p_scale (Fix noisy warning for uncalibrated q_scale/p_scale vllm-project/vllm#17414)
Add cutlass support for blackwell fp8 blockwise gemm (Add cutlass support for blackwell fp8 blockwise gemm vllm-project/vllm#14383)
[FEAT][ROCm]: Support AITER MLA on V1 Engine ([FEAT][ROCm]: Support AITER MLA on V1 Engine vllm-project/vllm#17523)
[V1][Structured Output] Update llguidance (>= 0.7.11) to avoid AttributeError (no StructTag) ([V1][Structured Output] Update llguidance (>= 0.7.11) to avoid AttributeError (no StructTag) vllm-project/vllm#17839)
[Attention] MLA move rotary embedding to cuda-graph region ([Attention] MLA move rotary embedding to cuda-graph region vllm-project/vllm#17668)
[BUGFIX]: return fast when request requires prompt logprobs ([BUGFIX]: return fast when request requires prompt logprobs vllm-project/vllm#17251)
[Docs] Add Slides from NYC Meetup ([Docs] Add Slides from NYC Meetup vllm-project/vllm#17879)
[Doc] Update several links in reasoning_outputs.md ([Doc] Update several links in reasoning_outputs.md vllm-project/vllm#17846)
[Doc] remove visible token in doc ([Doc] remove visible token in doc vllm-project/vllm#17884)
[Bugfix][ROCm] Fix AITER MLA V1 ([Bugfix][ROCm] Fix AITER MLA V1 vllm-project/vllm#17880)
[Bugfix][CPU] Fix broken AVX2 CPU TP support ([Bugfix][CPU] Fix broken AVX2 CPU TP support vllm-project/vllm#17252)
Fix Whisper crash caused by invalid max_num_batched_tokens config (Fix Whisper crash caused by invalid max_num_batched_tokens config vllm-project/vllm#17853)
Change top_k to be disabled with 0 (still accept -1 for now) (Change top_k to be disabled with 0 (still accept -1 for now) vllm-project/vllm#17773)
[Misc] add dify integration ([Misc] add dify integration vllm-project/vllm#17895)
[BugFix][AMD] Compatible patch for latest AITER(05/07/2025) ([BugFix][AMD] Compatible patch for latest AITER(05/07/2025) vllm-project/vllm#17864)
[v1] Move block management logic from KVCacheManager to SpecializedManager ([v1] Move block management logic from KVCacheManager to SpecializedManager vllm-project/vllm#17474)
[CI/Build] Automatically retry flaky tests ([CI/Build] Automatically retry flaky tests vllm-project/vllm#17856)
Revert "[BugFix][AMD] Compatible patch for latest AITER(05/07/2025)" (Revert "[BugFix][AMD] Compatible patch for latest AITER(05/07/2025)" vllm-project/vllm#17910)
[Misc] Add references in ray_serve_deepseek example ([Misc] Add references in ray_serve_deepseek example vllm-project/vllm#17907)
[Misc] Auto fallback to float16 for pre-Ampere GPUs when detected bfloat16 config ([Misc] Auto fallback to float16 for pre-Ampere GPUs when detected bfloat16 config vllm-project/vllm#17265)
Update CT WNA16MarlinMoE integration (Update CT WNA16MarlinMoE integration vllm-project/vllm#16666)
Handle error when str passed to /v1/audio/transcriptions (Handle error when str passed to /v1/audio/transcriptions vllm-project/vllm#17909)
Add option to use torch._inductor.standalone_compile (Add option to use torch._inductor.standalone_compile vllm-project/vllm#17057)
[V1][Spec Decoding] Include bonus tokens in mean acceptance length ([V1][Spec Decoding] Include bonus tokens in mean acceptance length vllm-project/vllm#17908)
Improve configs - the rest! (Improve configs - the rest! vllm-project/vllm#17562)
AMD conditional all test execution // new test groups (AMD conditional all test execution // new test groups vllm-project/vllm#17556)
[Hardware/NVIDIA/Kernel] Enable nvidia/DeepSeek-R1-FP4 Model ([Hardware/NVIDIA/Kernel] [Functional Enablement] [1/N] Enable nvidia/DeepSeek-R1-FP4 Model vllm-project/vllm#16362)
[V1][Spec Decoding] Log accumulated metrics after system goes idle ([V1][Spec Decoding] Log accumulated metrics after system goes idle vllm-project/vllm#17913)
fix broken test vllm:test_kernels - test_attention_selector.py::test_flash_attn (fix broken test vllm:test_kernels - test_attention_selector.py::test_flash_attn vllm-project/vllm#17873)
Add missing content type headers to /ping and /health ([Bug]: Multiple openai endpoint Missing Content-Type Header vllm-project/vllm#17036) (Add missing content type headers to /ping and /health (#17036) vllm-project/vllm#17786)
Don't default construct ModelConfig when default constructing VllmConfig (Don't default construct ModelConfig when default constructing VllmConfig vllm-project/vllm#17943)
[Misc] remove --model from vllm serve usage ([Misc] remove --model from vllm serve usage vllm-project/vllm#17944)
[v1] Pass BlockTable and KVCacheSpec to AttentionMetadataBuilders ([v1] Pass BlockTable and KVCacheSpec to AttentionMetadataBuilders vllm-project/vllm#17483)
[v1] Rename specialized_manager.py to single_type_kv_cache_manager.py ([v1] Rename specialized_manager.py to single_type_kv_cache_manager.py vllm-project/vllm#17946)
[Kernel] fp4 marlin kernel ([Kernel] fp4 marlin kernel vllm-project/vllm#17687)
[Bugfix] Add revision to transformers.Auto*.from_pretrained processors ([Bugfix] Add revision to transformers.Auto*.from_pretrained processors vllm-project/vllm#17948)
[Perf] Use small max_num_batched_tokens for A100 ([Perf] Use small max_num_batched_tokens for A100 vllm-project/vllm#17885)
fix amd triton mla path (fix amd triton mla path vllm-project/vllm#17871)
[Bugfix]: v1 engine - consider lora adapters in allowed_token_ids ([Bugfix]: v1 engine - consider lora adapters in allowed_token_ids vllm-project/vllm#17855)
[doc] update lora doc ([doc] update lora doc vllm-project/vllm#17936)
[Frontend] Add /classify endpoint ([Frontend] Add /classify endpoint vllm-project/vllm#17032)
[Misc] Add compressed-tensors NVFP4A16 emulation support ([Misc] Add compressed-tensors NVFP4A16 emulation support vllm-project/vllm#17914)
[FP8][ROCm][Attention] Enable FP8 KV cache on ROCm for V1 ([FP8][ROCm][Attention] Enable FP8 KV cache on ROCm for V1 vllm-project/vllm#17870)
[New Model]: nomic-embed-text-v2-moe ([New Model]: nomic-embed-text-v2-moe vllm-project/vllm#17785)
[Misc] not show --model in vllm serve --help ([Misc] not show --model in vllm serve --help vllm-project/vllm#16691)
[BugFix] [ROCm]: Bugfix and handle addition case of input for rocm_aiter_rms_norm ([BugFix] [ROCm]: Bugfix and handle addition case of input for rocm_aiter_rms_norm vllm-project/vllm#17857)
[BUG] [ROCm] [MLA] Fix variable name bug due to change in variable name in PR [v1] Pass BlockTable and KVCacheSpec to AttentionMetadataBuilders vllm-project/vllm#17483 ([BUG] [ROCm] [MLA] Fix variable name bug due to change in variable name in PR #17483 vllm-project/vllm#17961)
[Model] Broadcast Ovis2 implementation to fit Ovis1.6 ([Model] Broadcast Ovis2 implementation to fit Ovis1.6 vllm-project/vllm#17861)
[misc] add instructions on how to install nvshmem/pplx/deepep ([misc] add instructions on how to install nvshmem/pplx/deepep vllm-project/vllm#17964)
[Bugfix] validate grammar and throw 400 error instead of crashing the engine when xgrammar validation fails ([Bugfix] validate grammar and throw 400 error instead of crashing the engine when xgrammar validation fails vllm-project/vllm#17623)
[bugfix] fix the wrong parser ([bugfix] fix the wrong parser vllm-project/vllm#17958)
[Bugfix] Fix pydantic.errors.PydanticUserError ([Bugfix] Fix pydantic.errors.PydanticUserError vllm-project/vllm#17962)
[Bugfix][TPU] Use np array when updating cache slot_mapping ([Bugfix][TPU] Use np array when updating cache slot_mapping vllm-project/vllm#17971)
[Fix] Benchmark "EngineClient" has no attribute "model_config" ([Fix] Benchmark "EngineClient" has no attribute "model_config" vllm-project/vllm#17976)
[Feature] Support DeepSeekV3 Function Call ([Feature] Support DeepSeekV3 Function Call vllm-project/vllm#17784)
Correcting testcases in builkite job for IBM Power (Correcting testcases in builkite job for IBM Power vllm-project/vllm#17675)
[Misc] Improve modelscope import error ([Misc] Improve modelscope import error vllm-project/vllm#17983)
Initialize the delta tool call fields explicitly (Initialize the delta tool call fields explicitly vllm-project/vllm#17340)
[P/D] NIXL Integration ([P/D] NIXL Integration vllm-project/vllm#17751)
[Lora][Frontend]Add default local directory LoRA resolver plugin. ([Lora][Frontend]Add default local directory LoRA resolver plugin. vllm-project/vllm#16855)
Construct KVTransferConfig properly from Python instead of using JSON blobs without CLI (Construct KVTransferConfig properly from Python instead of using JSON blobs without CLI vllm-project/vllm#17994)
[CI/Build] Fix TPU V1 Test mixed use of & and && across tests ([CI/Build] Fix TPU V1 Test mixed use of & and && across tests vllm-project/vllm#17968)
[Core] Use platform-agnostic device control for DP engine core ([Core] Use platform-agnostic device control for DP engine core vllm-project/vllm#17245)
Enabling "Weight Loading Multiple GPU Test - Large Models" (Enabling "Weight Loading Multiple GPU Test - Large Models" vllm-project/vllm#18020)
[v1][KVCacheManager] Change prefix caching metric from counting blocks to counting tokens ([v1][KVCacheManager] Change prefix caching metric from counting blocks to counting tokens vllm-project/vllm#18003)
[Chore] Remove unused method ([Chore] Remove unused method vllm-project/vllm#18024)
Enable standard language model for torhc nightly (Enable standard language model for torhc nightly vllm-project/vllm#18004)
[CI] Make JSON output tests less likely to fail ([CI] Make JSON output tests less likely to fail vllm-project/vllm#17859)
[V1][Spec Decode] Eagle unit tests ([V1][Spec Decode] Eagle unit tests vllm-project/vllm#17350)
Fix FBGEMM integration ([Bugfix] Fix FBGEMM integration vllm-project/vllm#18002)
[Model] Support MiMo-7B inference with MTP ([Model] Support MiMo-7B inference with MTP vllm-project/vllm#17433)
Update some more deprecated type hinting (Update some more deprecated type hinting vllm-project/vllm#17998)
Use NVFP4 Marlin for CompressedTensorsW4A16Fp4 (Use NVFP4 Marlin for CompressedTensorsW4A16Fp4 vllm-project/vllm#18000)
Remove noisy warnings from SchedulerConfig (Remove noisy warnings from SchedulerConfig vllm-project/vllm#17995)
[ROCm] Skip tests for quantizations incompatible with ROCm ([ROCm] Skip tests for quantizations incompatible with ROCm vllm-project/vllm#17905)
Implements dual-chunk-flash-attn backend for dual chunk attention with sparse attention support (Implements dual-chunk-flash-attn backend for dual chunk attention with sparse attention support vllm-project/vllm#11844)
[Misc] Slight spelling modification ([Misc] Slight spelling modification vllm-project/vllm#18039)
[ROCm]: Fix build from source failure with gcc14 and ROCm 6.3 ([ROCm]: Fix build from source failure with gcc14 and ROCm 6.3 vllm-project/vllm#13779)
[Bugfix] Fixes for new marlin moe usage ([Bugfix] Fixes for new marlin moe usage vllm-project/vllm#18017)
[Bugfix] Avoid repeatedly creating dummy data during engine startup ([Bugfix] Avoid repeatedly creating dummy data during engine startup vllm-project/vllm#17935)
[Feature][V1] Support tool_choice: required when using Xgrammar as the StructuredOutputBackend. ([Feature][V1] Support tool_choice: required when using Xgrammar as the StructuredOutputBackend. vllm-project/vllm#17845)
cleanup invalid prints (cleanup invalid prints vllm-project/vllm#18050)
[BugFix] Fix 4-GPU RLHF tests ([BugFix] Fix 4-GPU RLHF tests vllm-project/vllm#18007)
Fix Broken macro for cutlass moe (Fix Broken macro for cutlass moe vllm-project/vllm#18049)
[v1][KVCacheManager] Avoid full cache hit by controlling max_length ([v1][KVCacheManager] Avoid full cache hit by controlling max_length vllm-project/vllm#17999)
[Bugfix][V1] Only get input embeddings w/ multi-modal models if first PP ([Bugfix][V1] Only get input embeddings w/ multi-modal models if first PP vllm-project/vllm#17916)
[BugFix] Set default random seed to 0 for V1 ([BugFix] Set default random seed to 0 for V1 vllm-project/vllm#17929)
[Bugfix] Fix marlin moe fallback logic for llama4 ([Bugfix] Fix marlin moe fallback logic for llama4 vllm-project/vllm#18042)
[Benchmarks] Refactor run_structured_output_benchmarks.sh ([Benchmarks] Refactor run_structured_output_benchmarks.sh vllm-project/vllm#17722)
Convert .buildkite to ruff format (Convert .buildkite to ruff format vllm-project/vllm#17656)
[Fix] check to make sure processor has chat templates ([Fix] check to make sure processor has chat templates vllm-project/vllm#18047)
[doc] add download/list/delete HF model CLI usage ([doc] add download/list/delete HF model CLI usage vllm-project/vllm#17940)
Update deprecated type hinting in model_executor/layers (Update deprecated type hinting in model_executor/layers vllm-project/vllm#18056)
Update deprecated type hinting in vllm/profiler (Update deprecated type hinting in vllm/profiler vllm-project/vllm#18057)
Update deprecated type hinting in vllm/transformers_utils (Update deprecated type hinting in vllm/transformers_utils vllm-project/vllm#18058)
[CI] Set token permissions for reminder comment CI job ([CI] Set token permissions for reminder comment CI job vllm-project/vllm#17728)
[CI] Add workflow permissions for helm CI job ([CI] Add workflow permissions for helm CI job vllm-project/vllm#17727)
[CI] Add token permissions for add-ready-label CI job ([CI] Add token permissions for add-ready-label CI job vllm-project/vllm#17730)
[CI] set token permissions for pre-commit CI job ([CI] set token permissions for pre-commit CI job vllm-project/vllm#17729)
[Bugfix] Fix entrypoints metrics tests ([Bugfix] Fix entrypoints metrics tests vllm-project/vllm#18063)
Convert benchmarks to ruff format (Convert benchmarks to ruff format vllm-project/vllm#18068)
Give auto-merge label workflow permission to add labels to issues (Give auto-merge label workflow permission to add labels to issues vllm-project/vllm#18078)
Update deprecated type hinting in vllm/compilation (Update deprecated type hinting in vllm/compilation vllm-project/vllm#18072)
Update deprecated type hinting in vllm/adapter_commons (Update deprecated type hinting in vllm/adapter_commons vllm-project/vllm#18073)
[V1] DP scale-out (2/N): Decouple engine process management and comms ([V1] DP scale-out (2/N): Decouple engine process management and comms vllm-project/vllm#15977)
[Docs] Expand security doc with firewall info ([Docs] Expand security doc with firewall info vllm-project/vllm#18081)
[FEAT] [ROCm]: Add AITER Block-Scaled GEMM Feature ([FEAT] [ROCm]: Add AITER Block-Scaled GEMM Feature vllm-project/vllm#14968)
[v1][KVCacheManager] pass num_new_computed_tokens to kv cache manager ([v1][KVCacheManager] pass num_new_computed_tokens to kv cache manager vllm-project/vllm#18001)
[Fix] Support CUDAGraph capture for encoder-decoder on ROCm ([Fix] Support CUDAGraph capture for encoder-decoder on ROCm vllm-project/vllm#18104)
[Hardware/NVIDIA/Modelopt] Fix modelopt forward method for v1 torch.compile ([Hardware/NVIDIA/Modelopt] Fix modelopt forward method for v1 torch.compile vllm-project/vllm#18101)
[P/D] Add some more debug logs to NixlConnector ([P/D] Add some more debug logs to NixlConnector vllm-project/vllm#18102)
[Misc] Remove unused numpy tensor ([Misc] Remove unused numpy tensor vllm-project/vllm#18084)
[Bug]: Fix S3 model/tokenizer path resolution ([Bug]: Fix S3 model/tokenizer path resolution vllm-project/vllm#18083)
[core][distributed] add ep group and all2all interface ([core][distributed] add ep group and all2all interface vllm-project/vllm#18077)
[Bugfix] Fix FP8 Marlin MoE and enable for compressed-tensors models ([Bugfix] Fix FP8 Marlin MoE and enable for compressed-tensors models vllm-project/vllm#18026)
[Bugfix][V1] Fix FlashInfer V1 backend using the wrong VllmConfig ([Bugfix][V1] Fix FlashInfer V1 backend using the wrong VllmConfig vllm-project/vllm#18086)
[FEAT] [ROCm] [V1]: Add AITER biased group topk for DeepSeekV3 ([FEAT] [ROCm] [V1]: Add AITER biased group topk for DeepSeekV3 vllm-project/vllm#17955)
[AMD][torch.compile] Enable silu+fp8_quant fusion for rocm ([AMD][torch.compile] Enable silu+fp8_quant fusion for rocm vllm-project/vllm#18082)
[BugFix][AMD] Compatible patch for AITER lib after 04/20 ([BugFix][AMD] Compatible patch for AITER lib after 04/20 vllm-project/vllm#17912)
Fix broken example: examples/offline_inference/profiling at scheduler_config (Fix broken example: examples/offline_inference/profiling at scheduler_config vllm-project/vllm#18117)
[Fix] Move "model_config" as keyword args in chat_utils.py ([Fix] Move "model_config" as keyword args in chat_utils.py vllm-project/vllm#18098)
[Bugfix] fix moe marlin topk_weight loading ([Bugfix] fix moe marlin topk_weight loading vllm-project/vllm#18080)
[Bugfix][Example] make lmcache v0 work. ([Bugfix][Example] make lmcache v0 work. vllm-project/vllm#18051)
[New Model]: support GTE NewModel ([New Model]: support GTE NewModel vllm-project/vllm#17986)
[Bugfix] Fix entrypoints audio test failure ([Bugfix] Fix entrypoints audio test failure vllm-project/vllm#18111)
[Model] Add packed_modules_mapping for Qwen3-MOE ([Model] Add packed_modules_mapping for Qwen3-MOE vllm-project/vllm#18118)
[Misc] replace does not exist model ([Misc] replace does not exist model vllm-project/vllm#18119)
[Bugfix] Fix QKVCrossParallelLinear::sync_weight_attrs for PyTorch compile ([Bugfix] Fix QKVCrossParallelLinear::sync_weight_attrs for PyTorch compile vllm-project/vllm#17844)
[FEAT] [ROCm]: Add AITER CK 2 Stages MoE support ([FEAT] [ROCm]: Add AITER CK 2 Stages MoE support vllm-project/vllm#17110)
[Bugfix] Fix LoRA test ([Bugfix] Fix LoRA test vllm-project/vllm#18123)
[Model] GritLM supports other attention backends ([Model] GritLM supports other attention backends vllm-project/vllm#18109)
[doc] add missing import ([doc] add missing import vllm-project/vllm#18133)
Update deprecated type hinting in vllm/lora (Update deprecated type hinting in vllm/lora vllm-project/vllm#18128)
Update deprecated type hinting in vllm/device_allocator and vllm/distributed (Update deprecated type hinting in vllm/device_allocator and vllm/distributed vllm-project/vllm#18126)
Update deprecated type hinting in platform, plugins, triton_utils, vllm_flash_attn (Update deprecated type hinting in platform, plugins, triton_utils, vllm_flash_attn vllm-project/vllm#18129)
[Bugfix] Fix chat utils tests ([Bugfix] Fix chat utils tests vllm-project/vllm#18139)
[KVConnector] Keep KVTransferParams as a dict ([KVConnector] Keep KVTransferParams as a dict vllm-project/vllm#18033)
[Doc] Update prefix cache metrics to counting tokens ([Doc] Update prefix cache metrics to counting tokens vllm-project/vllm#18138)
[V1][Spec Decode] Share input embedding of target model with EAGLE draft model to free ~1GB for llama 3 model ([V1][Spec Decode] Share input embedding of target model with EAGLE draft model to free ~1GB for llama 3 model vllm-project/vllm#17326)
Modularize fused experts and integrate PPLX kernels (Modularize fused experts and integrate PPLX kernels vllm-project/vllm#15956)
[CI] Disable Failing Tests ([CI] Disable Failing Tests vllm-project/vllm#18165)
[Frontend] decrease import time of vllm.multimodal ([Frontend] decrease import time of vllm.multimodal vllm-project/vllm#18031)
[Kernel] Have rotary embeddings support tensors ([Kernel] Have rotary embeddings support tensors vllm-project/vllm#18046)
[V1] Structured Outputs + Thinking compatibility ([V1] Structured Outputs + Thinking compatibility vllm-project/vllm#16577)
Add support for loading torchao models with AOPerModuleConfig (Add support for loading torchao models with AOPerModuleConfig vllm-project/vllm#17826)
[CI] Fix race condition in test_kv_cache_events test ([CI] Fix race condition in test_kv_cache_events test vllm-project/vllm#18169)
[V1] Support multiple kv connectors ([V1] Support multiple kv connectors vllm-project/vllm#17564)
Upload vllm index for the rc builds (Upload vllm index for the rc builds vllm-project/vllm#18173)
[Bugfix]: make most of test_openai_schema.py pass ([Bugfix]: make most of test_openai_schema.py pass vllm-project/vllm#17664)
[v1] Support multiple KV cache groups in GPU model runner ([v1] Support multiple KV cache groups in GPU model runner vllm-project/vllm#17945)
[V1][Metrics] Remove unused code ([V1][Metrics] Remove unused code vllm-project/vllm#18158)
[Chore] astral's ty ([Chore] astral's ty vllm-project/vllm#18116)
[Misc] add lobe-chat support ([Misc] add lobe-chat support vllm-project/vllm#18177)
[Fix][ROCm] Enforce eager for all encoder-decoder models on ROCm ([Fix][ROCm] Enforce eager for all encoder-decoder models on ROCm vllm-project/vllm#18154)
Update deprecated type hinting in models (Update deprecated type hinting in models vllm-project/vllm#18132)
[Bugfix] Fix fp8 tests for triton_unified_attention for Triton 3.3 ([Bugfix] Fix fp8 tests for triton_unified_attention for Triton 3.3 vllm-project/vllm#18013)
Support custom implementations of VideoLoader backends. (Support custom implementations of VideoLoader backends. vllm-project/vllm#18091)
[UT] Add ut for none hash ([UT] Add ut for none hash vllm-project/vllm#17892)
[Model] Allow the use of sliding window in Qwen2 ([Model] Allow the use of sliding window in Qwen2 vllm-project/vllm#17772)
[Bugfix] Fix FusedMoEPrepareAndFinalize for cuda-disalike backends ([Bugfix] Fix FusedMoEPrepareAndFinalize for cuda-disalike backends vllm-project/vllm#18178)
[CI] don't skip fixed test_kv_cache_events() ([CI] don't skip fixed test_kv_cache_events() vllm-project/vllm#18183)
[V1] Update zmq socket creation in nixl connector ([V1] Update zmq socket creation in nixl connector vllm-project/vllm#18148)
fix: typos (fix: typos vllm-project/vllm#18151)
Update deprecated type hinting in model_loader (Update deprecated type hinting in model_loader vllm-project/vllm#18130)
add tools into TokenizeChatRequest (add tools into TokenizeChatRequest vllm-project/vllm#18187)
[Kernel] [V1] Fix performance regression for triton unified attention ([Kernel] [V1] Fix performance regression for triton unified attention vllm-project/vllm#18161)
Adding "Basic Models Test" and "Multi-Modal Models Test (Extended) 3" in AMD Pipeline (Adding "Basic Models Test" and "Multi-Modal Models Test (Extended) 3" in AMD Pipeline vllm-project/vllm#18106)
Improve examples rendering in docs and GitHub (Improve examples rendering in docs and GitHub vllm-project/vllm#18203)
[Frontend] Fix chat template content format detection ([Frontend] Fix chat template content format detection vllm-project/vllm#18190)
[Bugfix]Change the exception thrown by call_hf_processor from RuntimeError to ValueError ([Bugfix]Change the exception thrown by call_hf_processor from RuntimeError to ValueError vllm-project/vllm#18181)
[Bugfix] [ROCm]: Remove assertion logic when using AITER fused moe in unquantizedMethod to reenable LLama4 BF16 ([Bugfix] [ROCm]: Remove assertion logic when using AITER fused moe in unquantizedMethod to reenable LLama4 BF16 vllm-project/vllm#18205)
[Misc] Avoid cuda graph log when sizes still match ([Misc] Avoid cuda graph log when sizes still match vllm-project/vllm#18202)
Adding "AMD: Tensorizer Test" to amdproduction. (Adding "AMD: Tensorizer Test" to amdproduction. vllm-project/vllm#18216)
[Bugfix] Fix test_eagle test ([Bugfix] Fix test_eagle test vllm-project/vllm#18223)
[Build] Allow shipping PTX on a per-file basis ([Build] Allow shipping PTX on a per-file basis vllm-project/vllm#18155)
[Bugfix] fix rotary embedding test for _get_padded_tensor_shape ([Bugfix] fix rotary embedding test for _get_padded_tensor_shape vllm-project/vllm#18229)
[Bugfix][ROCm] Use chunked_prefill_paged_decode as fallback for V1 attention on ROCm ([Bugfix][ROCm] Use chunked_prefill_paged_decode as fallback for V1 attention on ROCm vllm-project/vllm#18093)
[Model] vLLM v1 supports Medusa ([Model] vLLM v1 supports Medusa vllm-project/vllm#17956)
Allow users to pass arbitrary JSON keys from CLI (Allow users to pass arbitrary JSON keys from CLI vllm-project/vllm#18208)
Throw better error for when running into k8s service discovery issue (Throw better error for when running into k8s service discovery issue vllm-project/vllm#18209)
[Feature] Support Pipeline Parallism in torchrun SPMD offline inference for V1 ([Feature] Support Pipeline Parallism in torchrun SPMD offline inference for V1 vllm-project/vllm#17827)
[doc] fix multimodal example script ([doc] fix multimodal example script vllm-project/vllm#18089)
[PERF] Speed up Qwen2.5-VL model by speed up rotary position embedding ([PERF] Speed up Qwen2.5-VL model by speed up rotary position embedding const… vllm-project/vllm#17973)
[Misc] Add Ray Prometheus logger to V1 ([Misc] Add Ray Prometheus logger to V1 vllm-project/vllm#17925)
[Misc] Consolidate Audio tests into multimodal common generation tests ([Misc] Consolidate Audio tests into multimodal common generation tests vllm-project/vllm#18214)
use ceil_div in cutlass block scaling shape check (use ceil_div in cutlass block scaling shape check vllm-project/vllm#17918)
[Fix] Fix typo in resolve_hf_chat_template ([Fix] Fix typo in resolve_hf_chat_template vllm-project/vllm#18259)
[Model] Use autoweightloader for dbrx ([Model] Use autoweightloader for dbrx vllm-project/vllm#18251)
[Misc][MacOS] fix bfloat16 error ([Misc][MacOS] fix bfloat16 error vllm-project/vllm#18249)
[BugFix] Fix multi async save in MultiConnector ([BugFix] Fix multi async save in MultiConnector vllm-project/vllm#18246)
[BugFix] Fix ordering of KVConnector finished send/rcv sets ([BugFix] Fix ordering of KVConnector finished send/rcv sets vllm-project/vllm#18211)
[CI] Assign reviewer to mergify with changes to Tensorizer files ([CI] Assign reviewer to mergify with changes to Tensorizer files vllm-project/vllm#18278)
[Sampler] Adapt to FlashInfer 0.2.3 sampler API ([Sampler] Adapt to FlashInfer 0.2.3 sampler API vllm-project/vllm#15777)
[Bugfix] fix an illegal memory access was encountered of marlin kernel + act_order ([Bugfix] fix an illegal memory access was encountered of marlin kernel + act_order vllm-project/vllm#18245)
[Spec Decode] Don't fall back to V0 when spec decoding is enabled ([Spec Decode] Don't fall back to V0 when spec decoding is enabled vllm-project/vllm#18265)
[V1][P/D] Local attention optimization for NIXL ([V1][P/D] Local attention optimization for NIXL vllm-project/vllm#18170)
Move cli args docs to its own page ([Doc]: Arguments missing from docs page vllm-project/vllm#18228) (Move cli args docs to its own page (#18228) vllm-project/vllm#18264)
[Misc] reformat the collect-env output ([Misc] reformat the collect-env output vllm-project/vllm#18285)
[BugFix] Correct max_model_len derivation from config.json for Mistral format ([BugFix] Correct max_model_len derivation from config.json for Mistral format vllm-project/vllm#17937)
[P/D][V1] Support dynamic loading of external KV connector implementations ([P/D][V1] Support dynamic loading of external KV connector implementations vllm-project/vllm#18142)
[Hardware][TPU] Optionally import for TPU backend ([Hardware][TPU] Optionally import for TPU backend vllm-project/vllm#18269)
Update Dockerfile to build for Blackwell (Update Dockerfile to build for Blackwell vllm-project/vllm#18095)
Fixed build on ppc64le due to openssl conflicts (Fixed build on ppc64le due to openssl conflicts vllm-project/vllm#18262)
[Model] use AutoWeightsLoader for solar ([Model] use AutoWeightsLoader for solar vllm-project/vllm#18113)
[MISC] fix typo ([MISC] fix typo vllm-project/vllm#18305)
Support sequence parallelism combined with pipeline parallelism (Support sequence parallelism combined with pipeline parallelism vllm-project/vllm#18243)
[doc] update reasoning doc ([doc] update reasoning doc vllm-project/vllm#18306)
[Model] Use sigmoid for single-label classification ([Model] Use sigmoid for single-label classification vllm-project/vllm#18313)
Fix copy-paste error in phi4mm image processing (Fix copy-paste error in phi4mm image processing vllm-project/vllm#18315)
[Misc] add litellm integration ([Misc] add litellm integration vllm-project/vllm#18320)
[Doc] Add doc to explain the usage of Qwen3 thinking ([Doc] Add doc to explain the usage of Qwen3 thinking vllm-project/vllm#18291)
[Spec Decode][V0] Fix spec decode correctness test in V0 eagle/medusa ([Spec Decode][V0] Fix spec decode correctness test in V0 eagle/medusa vllm-project/vllm#18175)
Feature/vllm/input embedding completion api (Feature/vllm/input embedding completion api vllm-project/vllm#17590)
[Misc] extract parser.parse_args() ([Misc] extract parser.parse_args() vllm-project/vllm#18323)
[Build] Supports CUDA 12.6 and 11.8 after Blackwell Update ([Build] Supports CUDA 12.6 and 11.8 after Blackwell Update vllm-project/vllm#18316)
fix: Add type specifications for CLI arguments in tensorizer options (fix: Add type specifications for CLI arguments in tensorizer options vllm-project/vllm#18314)
[BugFix] [Vul] Add missing usedforsecurity=False in MD5 hashing to enable FIPS ([BugFix] [Vul] Add missing usedforsecurity=False in MD5 hashing to enable FIPS vllm-project/vllm#18319)
[Doc] Fix prompt embedding examples ([Doc] Fix prompt embedding examples vllm-project/vllm#18350)
[Doc] Move input-related docs to Features ([Doc] Move input-related docs to Features vllm-project/vllm#18353)
[BugFix] Fix handling of num_computed_tokens with connector ([BugFix] Fix handling of num_computed_tokens with connector vllm-project/vllm#18232)
[Quantization] Pool model support bitsandbytes ([Quantization] Pool model support bitsandbytes vllm-project/vllm#18087)
[Doc] Fix typo ([Doc] Fix typo vllm-project/vllm#18355)
[Frontend] add --quick option for vllm chat/complete ([Frontend] add --quick option for vllm chat/complete vllm-project/vllm#18297)
[Feature]Add support for models quantized with AutoRound ([Feature]Add support for models quantized with AutoRound vllm-project/vllm#17850)
Add files via uploadAdd fused MoE kernel tuning configs (fp8_w8a8) for DeepSeek V3/R1 on a single-node 8x NVIDIA H20 96GB setup (Add files via uploadAdd fused MoE kernel tuning configs (fp8_w8a8) for DeepSeek V3/R1 on a single-node 8x NVIDIA H20 96GB setup vllm-project/vllm#18337)
[Misc] Fix typo ([Misc] Fix typo vllm-project/vllm#18330)
Neuron up mistral (Neuron up mistral vllm-project/vllm#18222)
fix CUDA_check redefinition in use ceil_div in cutlass block scaling shape check vllm-project/vllm#17918 (fix CUDA_check redefinition in #17918 vllm-project/vllm#18287)
[neuron] fix authorization issue ([neuron] fix authorization issue vllm-project/vllm#18364)
[Misc] Allow AutoWeightsLoader to skip loading weights with specific substr in name ([Misc] Allow AutoWeightsLoader to skip loading weights with specific substr in name vllm-project/vllm#18358)
[Core] [Bugfix]: tensor parallel with prompt embeds ([Core] [Bugfix]: tensor parallel with prompt embeds vllm-project/vllm#18171)
[release] Change dockerhub username for TPU release ([release] Change dockerhub username for TPU release vllm-project/vllm#18389)
[Bugfix] fix adding bias twice in ipex GPTQ quantization ([Bugfix] fix adding bias twice in ipex GPTQ quantization vllm-project/vllm#18363)
[doc] update env variable export ([doc] update env variable export vllm-project/vllm#18391)
[Misc] Add LoRA code owner ([Misc] Add LoRA code owner vllm-project/vllm#18387)
Update cpu.txt (Update cpu.txt vllm-project/vllm#18398)
[CI] Add mteb testing to test the accuracy of the embedding model ([CI] Add mteb testing to test the accuracy of the embedding model vllm-project/vllm#17175)
[Bugfix] Fix MRoPE Errors in the Qwen-VL Model When Processing Pure Text ([Bugfix] Fix MRoPE Errors in the Qwen-VL Model When Processing Pure Text vllm-project/vllm#18407)
[Misc] refactor prompt embedding examples ([Misc] refactor prompt embedding examples vllm-project/vllm#18405)
[Minor] Rename quantization nvfp4 to modelopt_fp4 ([Minor] Rename quantization nvfp4 to modelopt_fp4 vllm-project/vllm#18356)
[Model] use AutoWeightsLoader for bloom ([Model] use AutoWeightsLoader for bloom vllm-project/vllm#18300)
[Kernel] update comment for KV shape in unified triton attn ([Kernel] update comment for KV shape in unified triton attn vllm-project/vllm#18099)
fix:Build torch wheel inline rather than picking from nightly (fix:Build torch wheel inline rather than picking from nightly vllm-project/vllm#18351)
[TPU] Re-enable the Pallas MoE kernel ([TPU] Re-enable the Pallas MoE kernel vllm-project/vllm#18025)
[Bugfix] config.head_dim is now explicitly set to None ([Bugfix] config.head_dim is now explicitly set to None vllm-project/vllm#18432)
[Bug] Fix moe_sum signature ([Bug] Fix moe_sum signature vllm-project/vllm#18440)
Revert "[Bugfix] Fix MRoPE Errors in the Qwen-VL Model When Processing Pure Text ([Bugfix] Fix MRoPE Errors in the Qwen-VL Model When Processing Pure Text vllm-project/vllm#18407)" (Revert "[Bugfix] Fix MRoPE Errors in the Qwen-VL Model When Processing Pure Text (#18407)" vllm-project/vllm#18456)
[Bugfix][Failing Test] Fix nixl connector test when promt size < block size ([Bugfix][Failing Test] Fix nixl connector test when promt size < block size vllm-project/vllm#18429)
[Misc] MultiConnector._connectors type ([Misc] MultiConnector._connectors type vllm-project/vllm#18423)
[Frontend] deprecate --device arg ([Frontend] deprecate --device arg vllm-project/vllm#18399)
[V1] Fix general plugins not loaded in engine for multiproc ([V1] Fix general plugins not loaded in engine for multiproc vllm-project/vllm#18326)
[Misc] refactor disaggregated-prefill-v1 example ([Misc] refactor disaggregated-prefill-v1 example vllm-project/vllm#18474)
[Bugfix][Failing Test] Fix test_events.py ([Bugfix][Failing Test] Fix test_events.py vllm-project/vllm#18460)
[MODEL] FalconH1 ([MODEL] FalconH1 vllm-project/vllm#18406)
[Doc] fix arg docstring in linear layers ([Doc] fix arg docstring in linear layers vllm-project/vllm#18410)
[Bugfix] Reduce moe_sum test size to avoid OOM ([Bugfix] Reduce moe_sum test size to avoid OOM vllm-project/vllm#18484)
[Build] fix Dockerfile shell ([Build] fix Dockerfile shell vllm-project/vllm#18402)
[Misc] Update deprecation message for --enable-reasoning ([Misc] Update deprecation message for --enable-reasoning vllm-project/vllm#18404)
[ROCm][Kernel][V1] Enable AMD Radeon GPU Custom Paged Attention on v1 ([ROCm][Kernel][V1] Enable AMD Radeon GPU Custom Paged Attention on v1 vllm-project/vllm#17004)
Revert "[v1] Support multiple KV cache groups in GPU model runner ([v1] Support multiple KV cache groups in GPU model runner vllm-project/vllm#17945) (Revert "[v1] Support multiple KV cache groups in GPU model runner (#17945) vllm-project/vllm#18459)
[FEAT][ROCm] Upgrade AITER MLA v1 backend ([FEAT][ROCm] Upgrade AITER MLA v1 backend vllm-project/vllm#18338)
[Bugfix] Consistent ascii handling in tool parsers ([Bugfix] Consistent ascii handling in tool parsers vllm-project/vllm#17704)
[FalconH1] Fix output dtype in RMSNorm fallback path for Falcon-H1 (e.g. 0.5B) ([FalconH1] Fix output dtype in RMSNorm fallback path for Falcon-H1 (e.g. 0.5B) vllm-project/vllm#18500)
[MISC] update project urls in pyproject.toml ([MISC] update project urls in pyproject.toml vllm-project/vllm#18519)
[CI] Fix race condition with StatelessProcessGroup.barrier ([CI] Fix race condition with StatelessProcessGroup.barrier vllm-project/vllm#18506)
Intialize io_thread_pool attribute in the beginning. (Intialize io_thread_pool attribute in the beginning. vllm-project/vllm#18331)
[Bugfix] Inconsistent token calculation compared to HF in llava family ([Bugfix] Inconsistent token calculation compared to HF in llava family vllm-project/vllm#18479)
[BugFix][DP] Send DP wave completion only from dp_rank==0 ([BugFix][DP] Send DP wave completion only from dp_rank==0 vllm-project/vllm#18502)
[Bugfix][Model] Make Olmo2Model weight loading return loaded weights ([Bugfix][Model] Make Olmo2Model weight loading return loaded weights vllm-project/vllm#18504)
[Bugfix] Fix LoRA test ([Bugfix] Fix LoRA test vllm-project/vllm#18518)
[Doc] Fix invalid JSON in example args ([Doc] Fix invalid JSON in example args vllm-project/vllm#18527)
[Neuron] Update Dockerfile.neuron to use latest neuron release (2.23) ([Neuron] Update Dockerfile.neuron to use latest neuron release (2.23) vllm-project/vllm#18512)
Update default neuron config for speculation (Update default neuron config for speculation vllm-project/vllm#18274)
Order sequence ids + config update to support specifying custom quantization layers (Order sequence ids + config update to support specifying custom quantization layers vllm-project/vllm#18279)
[Bugfix] Fix MRoPE Errors in the Qwen-VL Model When Processing Pure Text ([Bugfix] Fix MRoPE Errors in the Qwen-VL Model When Processing Pure Text vllm-project/vllm#18526)
[Bugfix] Add kwargs to RequestOutput init to be forward compatible ([Bugfix] Add kwargs to RequestOutput __init__ to be forward compatible vllm-project/vllm#18513)
[CI/Build] Update bamba test model location ([CI/Build] Update bamba test model location vllm-project/vllm#18544)
[Doc] Support --stream arg in openai_completion_client.py script ([Doc] Support --stream arg in openai_completion_client.py script vllm-project/vllm#18388)
[Bugfix] Use random hidden states in dummy sampler run ([Bugfix] Use random hidden states in dummy sampler run vllm-project/vllm#18543)
[Doc] Add stream flag for chat completion example ([Doc] Add stream flag for chat completion example vllm-project/vllm#18524)
[BugFix][CPU] Fix x86 SHM distributed module initialization ([BugFix][CPU] Fix x86 SHM distributed module initialization vllm-project/vllm#18536)
[Misc] improve Automatic Prefix Caching example ([Misc] improve Automatic Prefix Caching example vllm-project/vllm#18554)
[Misc] Call ndarray.tobytes() directly instead of ndarray.data.tobytes() ([Misc] Call ndarray.tobytes() directly instead of ndarray.data.tobytes() vllm-project/vllm#18347)
[Bugfix] make test_openai_schema.py pass ([Bugfix] make test_openai_schema.py pass vllm-project/vllm#18224)
[Platform] Move platform check to right place ([Platform] Move platform check to right place vllm-project/vllm#18470)
[Compile][Platform] Make PiecewiseBackend pluggable and extendable ([Compile][Platform] Make PiecewiseBackend pluggable and extendable vllm-project/vllm#18076)
[Build/CI] Fix CUDA 11.8 build ([Build/CI] Fix CUDA 11.8 build vllm-project/vllm#17679)
[Tool] Add NIXL installation script ([Tool] Add NIXL installation script vllm-project/vllm#18172)
[V1][Spec Decode][Bugfix] Load quantize weights for EAGLE ([V1][Spec Decode][Bugfix] Load quantize weights for EAGLE vllm-project/vllm#18290)
[Frontend][Bug Fix] Update llama4 pythonic jinja template and llama4_pythonic parser ([Frontend][Bug Fix] Update llama4 pythonic jinja template and llama4_pythonic parser vllm-project/vllm#17917)
[Frontend] [Core] Add Tensorizer support for V1, LoRA adapter serialization and deserialization ([Frontend] [Core] Add Tensorizer support for V1, LoRA adapter serialization and deserialization vllm-project/vllm#17926)
[AMD] [P/D] Compute num gpus for ROCm correctly in run_accuracy_test.sh ([AMD] [P/D] Compute num gpus for ROCm correctly in run_accuracy_test.sh vllm-project/vllm#18568)
Re-submit: Fix: Proper RGBA -> RGB conversion for PIL images. (Re-submit: Fix: Proper RGBA -> RGB conversion for PIL images. vllm-project/vllm#18569)
[V1][Spec Decoding] Use model_loader.get_model() to load models ([V1][Spec Decoding] Use model_loader.get_model() to load models vllm-project/vllm#18273)
Enable hybrid attention models for Transformers backend (Enable interleaved sliding window attention models for Transformers backend vllm-project/vllm#18494)
[Misc] refactor: simplify input validation and num_requests handling in _convert_v1_inputs ([Misc] refactor: simplify input validation and num_requests handling in _convert_v1_inputs vllm-project/vllm#18482)
[BugFix] Increase TP execute_model timeout ([BugFix] Increase TP execute_model timeout vllm-project/vllm#18558)
[Bugfix] Set KVTransferConfig.engine_id in post_init ([Bugfix] Set KVTransferConfig.engine_id in post_init vllm-project/vllm#18576)
[Spec Decode] Make EAGLE3 draft token ID mapping optional ([Spec Decode] Make EAGLE3 draft token ID mapping optional vllm-project/vllm#18488)
[Neuron] Remove bypass on EAGLEConfig and add a test ([Neuron] Remove bypass on EAGLEConfig and add a test vllm-project/vllm#18514)
[Bugfix][Benchmarks] Fix a benchmark of deepspeed-mii backend to use api_key ([Bugfix][Benchmarks] Fix a benchmark of deepspeed-mii backend to use api_key vllm-project/vllm#17291)
[Misc] Replace cuda hard code with current_platform ([Misc] Replace cuda hard code with current_platform vllm-project/vllm#16983)
[Hardware] correct method signatures for HPU,ROCm,XPU ([Hardware] correct method signatures for HPU,ROCm,XPU vllm-project/vllm#18551)
[V1] [Bugfix] eagle bugfix and enable correct lm_head for multimodal ([V1] [Bugfix] eagle bugfix and enable correct lm_head for multimodal vllm-project/vllm#18034)
[Feature]Add async tensor parallelism using compilation pass ([Feature]Add async tensor parallelism using compilation pass vllm-project/vllm#17882)
[Doc] Update quickstart and install for cu128 using --torch-backend=auto ([Doc] Update quickstart and install for cu128 using --torch-backend=auto vllm-project/vllm#18505)
[Feature][V1]: suupports cached_tokens in response usage ([Feature][V1]: suupports cached_tokens in response usage vllm-project/vllm#18149)
[Bugfix] Add half type support in reshape_and_cache_cpu_impl on x86 cpu platform ([Bugfix] Add half type support in reshape_and_cache_cpu_impl on x86 cpu platform vllm-project/vllm#18430)
Migrate docs from Sphinx to MkDocs (Migrate docs from Sphinx to MkDocs vllm-project/vllm#18145)
Revert "[V1] [Bugfix] eagle bugfix and enable correct lm_head for multimodal ([V1] [Bugfix] eagle bugfix and enable correct lm_head for multimodal vllm-project/vllm#18034)" (Revert "[V1] [Bugfix] eagle bugfix and enable correct lm_head for multimodal (#18034)" vllm-project/vllm#18600)
[Bugfix][Model] Fix baichuan model loader for tp ([Bugfix][Model] Fix baichuan model loader for tp vllm-project/vllm#18597)
[V0][Bugfix] Fix parallel sampling performance regression when guided decoding is enabled ([V0][Bugfix] Fix parallel sampling performance regression when guided decoding is enabled vllm-project/vllm#17731)
Add myself as docs code owner (Add myself as docs code owner vllm-project/vllm#18605)
[Hardware][CPU] Update intel_extension_for_pytorch 2.7.0 and move to requirements/cpu.txt ([Hardware][CPU] Update intel_extension_for_pytorch 2.7.0 and move to requirements/cpu.txt vllm-project/vllm#18542)
[CI] fix kv_cache_type argument ([CI] fix kv_cache_type argument vllm-project/vllm#18594)
[Doc] Fix indent of contributing to vllm ([Doc] Fix indent of contributing to vllm vllm-project/vllm#18611)
Replace {func} with mkdocs style links (Replace {func} with mkdocs style links vllm-project/vllm#18610)
[CI/Build] Fix V1 flag being set in entrypoints tests ([CI/Build] Fix V1 flag being set in entrypoints tests vllm-project/vllm#18598)
Fix examples with code blocks in docs (Fix examples with code blocks in docs vllm-project/vllm#18609)
[Bugfix] Fix transformers model impl ignored for mixtral quant ([Bugfix] Fix transformers model impl ignored for mixtral quant vllm-project/vllm#18602)
Include private attributes in API documentation (Include private attributes in API documentation vllm-project/vllm#18614)
[Misc] add Haystack integration ([Misc] add Haystack integration vllm-project/vllm#18601)
[Bugfix][Build/CI] Fixup CUDA compiler version check for CUDA_SUPPORTED_ARCHS ([Bugfix][Build/CI] Fixup CUDA compiler version check for CUDA_SUPPORTED_ARCHS vllm-project/vllm#18579)
[Doc] Fix markdown list indentation for MkDocs rendering ([Doc] Fix markdown list indentation for MkDocs rendering vllm-project/vllm#18620)
[Doc] Use a different color for the announcement ([Doc] Use a different color for the announcement vllm-project/vllm#18616)
Refactor pplx init logic to make it modular (prepare for deepep) (Refactor pplx init logic to make it modular (prepare for deepep) vllm-project/vllm#18200)
Fix figures in design doc (Fix figures in design doc vllm-project/vllm#18612)
[Docs] Change mkdocs to not use directory urls ([Docs] Change mkdocs to not use directory urls vllm-project/vllm#18622)
[v1] Redo "Support multiple KV cache groups in GPU model runner ([v1] Support multiple KV cache groups in GPU model runner vllm-project/vllm#17945)" ([v1] Redo "Support multiple KV cache groups in GPU model runner (#17945)" vllm-project/vllm#18593)
[Doc] fix list formatting ([Doc] fix list formatting vllm-project/vllm#18624)
[Doc] Fix top-level API links/docs ([Doc] Fix top-level API links/docs vllm-project/vllm#18621)
[Doc] Avoid documenting dynamic / internal modules ([Doc] Avoid documenting dynamic / internal modules vllm-project/vllm#18626)
[Doc] Fix broken links and unlinked docs, add shortcuts to home sidebar ([Doc] Fix broken links and unlinked docs, add shortcuts to home sidebar vllm-project/vllm#18627)
[V1] Support Deepseek MTP ([V1] Support Deepseek MTP vllm-project/vllm#18435)
Use prebuilt FlashInfer x86_64 PyTorch 2.7 CUDA 12.8 wheel for CI (Use prebuilt FlashInfer x86_64 PyTorch 2.7 CUDA 12.8 wheel for CI vllm-project/vllm#18537)
[CI] Enable test_initialization to run on V1 ([CI] Enable test_initialization to run on V1 vllm-project/vllm#16736)
[Doc] Update references to doc files ([Doc] Update references to doc files vllm-project/vllm#18637)
[ModelOpt] Introduce VLLM_MAX_TOKENS_PER_EXPERT_FP4_MOE env var to control blockscale tensor allocation ([ModelOpt] Introduce VLLM_MAX_TOKENS_PER_EXPERT_FP4_MOE env var to control blockscale tensor allocation vllm-project/vllm#18160)
[Bugfix] Migrate to REGEX Library to prevent catastrophic backtracking ([Bugfix] Migrate to REGEX Library to prevent catastrophic backtracking vllm-project/vllm#18454)
[Bugfix][Nixl] Fix Preemption Bug ([Bugfix][Nixl] Fix Preemption Bug vllm-project/vllm#18631)
config.py: Clarify that only local GGUF checkpoints are supported. (config.py: Clarify that only local GGUF checkpoints are supported. vllm-project/vllm#18623)
FIX MOE issue in AutoRound format (FIX MOE issue in AutoRound format vllm-project/vllm#18586)
[V1][Spec Decode] Small refactors to improve eagle bookkeeping performance ([V1][Spec Decode] Small refactors to improve eagle bookkeeping performance vllm-project/vllm#18424)
[Frontend] improve vllm serve --help display ([Frontend] improve vllm serve --help display vllm-project/vllm#18643)
[Model] Add support for Qwen2.5-Omni-7B-AWQ (Qwen2_5OmniForConditionalGeneration) ([Model] Add support for Qwen2.5-Omni-7B-AWQ (Qwen2_5OmniForConditionalGeneration) vllm-project/vllm#18647)
[V1][Spec Decode] Support multi-layer eagle draft model ([V1][Spec Decode] Support multi-layer eagle draft model vllm-project/vllm#18030)
[Doc] Update README links, mark external links ([Doc] Update README links, mark external links vllm-project/vllm#18635)
[MISC][pre-commit] Add pre-commit check for triton import ([MISC][pre-commit] Add pre-commit check for triton import vllm-project/vllm#17716)
[Doc] Fix indentation problems in V0 Paged Attention docs ([Doc] Fix indentation problems in V0 Paged Attention docs vllm-project/vllm#18659)
[Doc] Add community links ([Doc] Add community links vllm-project/vllm#18657)
[Model] use AutoWeightsLoader for gpt2 ([Model] use AutoWeightsLoader for gpt2 vllm-project/vllm#18625)
[Doc] Reorganize user guide ([Doc] Reorganize user guide vllm-project/vllm#18661)
[CI/Build] chmod +x to cleanup_pr_body.sh ([CI/Build] chmod +x to cleanup_pr_body.sh vllm-project/vllm#18650)
[MISC] typo fix and clean import ([MISC] typo fix and clean import vllm-project/vllm#18664)
[BugFix] Fix import error for fused_moe ([BugFix] Fix import error for fused_moe vllm-project/vllm#18642)
[CI] enforce import regex instead of re ([CI] enforce import regex instead of re vllm-project/vllm#18665)
fix(regression): clone from reference items (fix(regression): clone from reference items vllm-project/vllm#18662)
[CI/Build] fix permission denied issue ([CI/Build] fix permission denied issue vllm-project/vllm#18645)
[BugFix][Spec Decode] Improve Prefix Caching Logic in Speculative Decoding ([BugFix][Spec Decode] Improve Prefix Caching Logic in Speculative Decoding vllm-project/vllm#18668)
[V1] Fix _pickle.PicklingError: Can't pickle <class 'transformers_modules.deepseek-ai.DeepSeek-V2-Lite... ([V1] Fix _pickle.PicklingError: Can't pickle <class 'transformers_modules.deepseek-ai.DeepSeek-V2-Lite... vllm-project/vllm#18640)
[MISC] correct signature for LoaderFunction ([MISC] correct signature for LoaderFunction vllm-project/vllm#18670)
[Misc]Replace cuda hard code with current_platform in Ray ([Misc]Replace cuda hard code with current_platform in Ray vllm-project/vllm#14668)
[Misc][ModelScope] Change to use runtime VLLM_USE_MODELSCOPE ([Misc][ModelScope] Change to use runtime VLLM_USE_MODELSCOPE vllm-project/vllm#18655)
[VLM] Initialize video input support for InternVL models ([VLM] Initialize video input support for InternVL models vllm-project/vllm#18499)
Speed up the kernels/quantization/ tests (Speed up the kernels/quantization/ tests vllm-project/vllm#18669)
[BUGFIX] catch subclass first for try...except ([BUGFIX] catch subclass first for try...except vllm-project/vllm#18672)
[Misc] Reduce logs on startup ([Misc] Reduce logs on startup vllm-project/vllm#18649)
[doc] fix broken links ([doc] fix broken links vllm-project/vllm#18671)
[doc] improve readability ([doc] improve readability vllm-project/vllm#18675)
[Bugfix] Fix cpu usage and cache hit stats reporting on cpu environment ([Bugfix] Fix cpu usage and cache hit stats reporting on cpu environment vllm-project/vllm#18674)
[CI/build] fix no regex ([CI/build] fix no regex vllm-project/vllm#18676)
[Misc] small improve ([Misc] small improve vllm-project/vllm#18680)
[Bugfix] Fix profiling dummy data for Pixtral ([Bugfix] Fix profiling dummy data for Pixtral vllm-project/vllm#18677)
[Core][Multimodal] Convert PIL Image to array without data copy when hashing ([Core][Multimodal] Convert PIL Image to array without data copy when hashing vllm-project/vllm#18682)
[CI/Build][Doc] Update gte-Qwen2-1.5B-instruct usage ([CI/Build][Doc] Update gte-Qwen2-1.5B-instruct usage vllm-project/vllm#18683)
[Misc] Fixed the abnormally high TTFT issue in the PD disaggregation example ([Misc] Fixed the abnormally high TTFT issue in the PD disaggregation example vllm-project/vllm#18644)
refactor: simplify request handler, use positive condition check for handler assignment (refactor: simplify request handler, use positive condition check for handler assignment vllm-project/vllm#18690)
[Bugfix] Fix the lm_head in gpt_bigcode in lora mode ([Bugfix] Fix the lm_head in gpt_bigcode in lora mode vllm-project/vllm#6357)
[CI] add missing argument ([CI] add missing argument vllm-project/vllm#18694)
[GH] Add issue template for reporting CI failures ([GH] Add issue template for reporting CI failures vllm-project/vllm#18696)
[Doc] Fix issue template format ([Doc] Fix issue template format vllm-project/vllm#18699)
[Bugfix] Fix Mistral-format models with sliding window ([Bugfix] Fix Mistral-format models with sliding window vllm-project/vllm#18693)
[CI/Build] Replace math.isclose with pytest.approx ([CI/Build] Replace math.isclose with pytest.approx vllm-project/vllm#18703)
[CI] fix dump_input for str type ([CI] fix dump_input for str type vllm-project/vllm#18697)
[Model] Add support for YARN in NemotronNAS models ([Model] Add support for YARN in NemotronNAS models vllm-project/vllm#18427)
[CI/Build] Split pooling and generation extended language models tests in CI ([CI/Build] Split pooling and generation extended language models tests in CI vllm-project/vllm#18705)
[Hardware][Intel-Gaudi] [CI/Build] Add tensor parallel size = 2 test to HPU CI ([Hardware][Intel-Gaudi] [CI/Build] Add tensor parallel size = 2 test to HPU CI vllm-project/vllm#18709)
[Misc] add AutoGen integration ([Misc] add AutoGen integration vllm-project/vllm#18712)
[Bugfix]: handle hf-xet CAS error when loading Qwen3 weights in vLLM ([Bugfix]: handle hf-xet CAS error when loading Qwen3 weights in vLLM vllm-project/vllm#18701)
[Doc] Improve API docs ([Doc] Improve API docs vllm-project/vllm#18713)
[Doc] Move examples and further reorganize user guide ([Doc] Move examples and further reorganize user guide vllm-project/vllm#18666)
[Bugfix] Fix Llama GGUF initialization ([Bugfix] Fix Llama GGUF initialization vllm-project/vllm#18717)
[V1][Sampler] Improve performance of FlashInfer sampling by sampling logits instead of probs ([V1][Sampler] Improve performance of FlashInfer sampling by sampling logits instead of probs vllm-project/vllm#18608)
Convert examples to ruff-format (Convert examples to ruff-format vllm-project/vllm#18400)
[Model][Gemma3] Simplify image input validation ([Model][Gemma3] Simplify image input validation vllm-project/vllm#18710)
[Misc] improve web section group title display ([Misc] improve web section group title display vllm-project/vllm#18684)
[V1][Quantization] Add CUDA graph compatible v1 GGUF support ([V1][Quantization] Add CUDA graph compatible v1 GGUF support vllm-project/vllm#18646)
[Model][Gemma3] Cast image pixel values already on CPU ([Model][Gemma3] Cast image pixel values already on CPU vllm-project/vllm#18732)
[FEAT] [ROCm] Upgrade AITER Fused MoE kernels. ([FEAT] [ROCm] Upgrade AITER Fused MoE kernels. vllm-project/vllm#18271)
[Doc] Update OOT model docs ([Doc] Update OOT model docs vllm-project/vllm#18742)
[Doc] Update reproducibility doc and example ([Doc] Update reproducibility doc and example vllm-project/vllm#18741)
[Misc] improve docs ([Misc] improve docs vllm-project/vllm#18734)
feat(rocm-support): support mamba2 on rocm (feat(rocm-support): support mamba2 on rocm vllm-project/vllm#18565)
[Hardware][Intel-Gaudi] [CI/Build] Fix multiple containers using the same name in run-hpu-test.sh ([Hardware][Intel-Gaudi] [CI/Build] Fix multiple containers using the same name in run-hpu-test.sh vllm-project/vllm#18752)
[Doc] cleanup deprecated flag for doc ([Doc] cleanup deprecated flag for doc vllm-project/vllm#18715)
Minor fix about MooncakeStoreConnector (Minor fix about MooncakeStoreConnector vllm-project/vllm#18721)
[Build] fix cpu build missing libtbbmalloc.so ([Build] fix cpu build missing libtbbmalloc.so vllm-project/vllm#18744)
[BUG FIX] minicpm ([BUG FIX] minicpm vllm-project/vllm#18739)
[Doc] Convert Sphinx directives ( {class}, {meth}, {attr}, ...) to MkDocs format for better documentation linking ([Doc] Convert Sphinx directives ( {class}, {meth}, {attr}, ...) to MkDocs format for better documentation linking vllm-project/vllm#18663)
[CI/Build] Remove imports of built-in re ([CI/Build] Remove imports of built-in re vllm-project/vllm#18750)
[V1][Metrics] Add API for accessing in-memory Prometheus metrics ([V1][Metrics] Add API for accessing in-memory Prometheus metrics vllm-project/vllm#17010)
Disable prefix cache by default for benchmark (#18639)
optimize get_kv_cache_torch_dtype (#18531)
[Core] Automatically cast multi-modal input dtype (#18756)
[Bugfix] Mistral tool calling when content is list (#18729)
[CI/Build] [TPU] Fix TPU CI exit code (#18282)
[Neuron] Support quantization on neuron (#18283)
Support datasets in vllm bench serve and sync with benchmark_[serving,datasets].py (#18566)
[Bugfix] Disable prefix caching by default for benchmark (#18771)
[Build] Fixes for CMake install (#18570)
[Core] Improve Tensor serialisation (#18774)
[rocm] Fix wrong attention log (#18764)
[Bugfix] Fix nomic max_model_len (#18755)
[Bugfix]: correctly propagate errors message caught at the chat_templating step to the client (#18769)
[V1] fix torch profiling for V1 offline scenarios (#18445)
[V1] [Bugfix] eagle bugfix and enable correct lm_head for multimodal (2) (#18781)
[Bugfix][FailingTest]Fix test_model_load_with_params.py (#18758)
[Deprecation] Require overriding get_dummy_text and get_dummy_mm_data (#18796)
[Deprecation] Remove unused sync methods in async_timeout (#18792)
[Deprecation] Remove fallbacks for Embeddings API (#18795)
[CI] improve embed testing (#18747)
Fix PiecewiseCompileInterpreter (#17338)
[BugFix] FA2 MLA Accuracy Issue (#18807)
[Platform][Dist] Make torch distributed process group extendable (#18763)
Enable Pydantic mypy checks and convert configs to Pydantic dataclasses (#17599)
[Frontend] add run batch to CLI (#18804)
decrement server_load on listen for disconnect (#18784)
[Core] Add Lora Support to Beam Search (#18346)
[Chore] update ty configuration (#18839)
[Misc] fix olmoe model layer can't laod in tp gt 1 (#18828)
[V1][Metrics] Remove metrics that were deprecated in 0.8 (#18837)
[Chore][Spec Decode] Update check NoneType instead of assigning variables (#18836)
[Hardware][TPU][V1] Multi-LoRA Optimisations for the V1 TPU backend (#15655)
Remove checks for None for fields which should never be None (#17985)
[Core] Enable CUDA graphs for DP + All2All kernels (#18724)
[Bugfix][ROCm] fix the power of 2 exception from triton_unified_attention.py when running llama4 models and unit test fix (#18100)
Prevent the cross-encoder logic from being applied to classification tasks (#18838)
Add ability to use CUDAGraphs with use_inductor=False (#17345)
[Bugfix][TPU] fix moe custom kernel import (#18853)
[Doc][Neuron] Update documentation for Neuron (#18868)
Skip device and quant Pydantic validation to make plugin device work (#18843)
Fixes a dead link in nightly benchmark readme (#18856)
[Neuron] Add multi-LoRA support for Neuron. (#18284)
[LoRA] Add LoRA support for InternVL (#18842)
[Doc] Remove redundant spaces from compatibility_matrix.md (#18891)
[doc] add CLI doc (#18871)
[Bugfix] Fix misleading information in the documentation (#18845)
[Misc] Replace TODO in serving transcription (#18895)
[Bugfix] Ensure tensors are contiguous during serialisation (#18860)
[BugFix] Update pydantic to fix error on python 3.10 (#18852)
Fix an error in dummy weight loading for quantization models (#18855)
[Misc][Tools][Benchmark] Add benchmark_serving supports for llama.cpp. (#18692)
[Doc] Fix codeblocks formatting in LoRA adapters documentation (#18907)
[Bugfix] Fix the failing gte embedding test (#18720)
[Attention][V1] Toggle for v1 attention backend (#18275)
[ROCm][V0][Attention] Revert to the previous FA triton kernel (#18226)
[Deprecation] Disallow pos-args other than model when initializing LLM (#18802)
[Misc] Remove duplicate init for self.vllm_config (#18896)
[V1] Allocate kv_cache with stride order for V1 (#18775)
[BugFix] Make DP work with connector-delayed new requests (#18559)
[P/D] NixlConnector DP fixes (#18903)
Use standalone_compile by default in torch >= 2.8.0 (#18846)
[TPU] remove transpose ops in moe kernel (#18923)
[Bugfix] Fix PP default fallback behavior for V1 (#18915)
[Misc] Update type annotation for rotary embedding base (#18914)
[TPU][CI/CD] Clean up docker for TPU tests. (#18926)
improve the robustness of parsing vlms config in AutoRound (#18894)
[Bugfix] Consistent ascii handling in tool parsers (#18883)
[Model] Use AutoWeightsLoader for mamba2 (#18918)
[docs] fix: fix markdown syntax (#18927)
[ROCm] Remove unnecessary assertion of max_model_len in ROCM_AITER_MLA attention backend. (#18938)
[Bugfix] Remove NVFP4 scales assertions to fix load_format=dummy (#18861)
[Deprecation] Remove mean pooling default for Qwen2EmbeddingModel (#18913)
[Misc]Fix benchmarks/README.md for speculative decoding (#18897)
[doc] add mkdocs doc (#18930)
[Model] Use in-place adds in SigLIP (#18922)
[Bugfix][Failing Test] Fix test_vllm_port.py (#18618)
[Misc]Fix typo (#18947)
[Bugfix][TPU] Fix tpu model runner testcase failure (#18810)
[CI/Build] remove regex from build dependencies (#18945)
[Feature] minicpm eagle support (#18943)
[doc] show the count for fork and watch (#18950)
[Docs] Update SECURITY.md with link to our security guide (#18961)
Improve "failed to get the hash of the compiled graph" error (#18956)
[Perf] API-server scaleout with many-to-many server-engine comms (#17546)
Benchmark script for fp8 vs bf16 gemm (#17126)
[VLM] Add PP support and fix GPTQ inference for Ovis models (#18958)
[Misc] add group_size is -1 in awq quantization (#18910)
Tool parser regex timeout handling (#18960)
[Docs] Correct multiprocessing design doc (#18964)
create util function for batched arange (#18937)
[Frontend] Add rerank support to run_batch endpoint (#16278)
[Misc] Fix estimated max model len msg (#18966)
[Bugfix]: Fix the incompatibility issue with Structured Outputs when Thinking is disabled (#18879)
fix security issue of logging llm output (#18980)
[Neuron] Add Multi-Modal model support for Neuron (#18921)
[doc] fix the list rendering issue - security.md (#18982)
[BugFix] Pydantic part 2 (#18911)
[FEAT][ROCm] Add AITER grouped topk for DeepSeekV2 (#18825)
[Bugfix] Fix for issue 17396 (#18773)
[ROCm][Kernel] Add gfx950 support for skinny gemms (#18010)
[P/D] NixlConnector use cache device index for memory registration (#18969)
[BugFix] Fix multi-node offline data-parallel (#18981)
[Misc] add return token strs for tokenize (#18941)
[Misc][Benchmark] Add support for CustomDataset (#18511)
[Bugfix] Fix EAGLE3 broken logits (#18909)
[Core] Rework dtype resolution (#18751)
[LoRA] Support dynamically initialize packed_modules_mapping for VLM with arbitrary components (#18987)
[doc] small fix - mkdocs (#18996)
Let max_num_batched_tokens use human_readable_int for large numbers (#18968)
[BugFix] fix data parallel construct ipv6 url addres (#18991)
[BugFix] Fix incorrect metrics shutdown error log message (#18992)
[doc] wrong output (#19000)
[Misc] reuse num_tokens_across_dp of get_dp_padding to avoid unnecessary dp all reduce in set_forward_context (#18935)
[Bugfix][Nixl] Fix DP Metadata Handshake (#19008)
[Core] Support inplace model weights loading (#18745)
[doc] add pytest tips (#19010)
[Model] enable data parallel for Llama4 vision encoder (#18368)
[Frontend] enable custom logging for the uvicorn server (OpenAI API server) (#18403)
[Bugfix][Model] Attempt to fix eagle in V0. (#18978)
add an absolute path for run.sh (#18258)
[Hardware][TPU] Initial support of model parallelism with single worker using SPMD (#18011)
[Doc] Remove duplicate TOCs during MkDocs migration (#19021)
[Bugfix][EP+DP] Use pplx-kernel internode instead of intranode (#19034)
Adding "LoRA Test %N" to AMD production tests (#18929)
[CPU][CI] Re-enable the CPU CI tests (#19046)
[ROCm][Build] Clean up the ROCm build (#19040)
[V1] Support DP with Ray (#18779)
Add tarsier model support (#18985)
[bugfix] small fix logic issue (#18999)
Reduce logs in CLI scripts and plugin loader (#18970)
[Bugfix] Use cmake 3.26.1 instead of 3.26 to avoid build failure (#19019)
[v1][KVCacheManager] Rename BlockHashType to BlockHash (#19015)
Update docker docs with ARM CUDA cross-compile (#19037)
[Doc] Add InternVL LoRA support (#19055)
[Misc] Update WeightsMapper for qwen2-vl/qwen2.5-vl (#19054)
[Doc] Update V1 user guide for embedding and enc-dec models (#19060)
[doc] clarify windows support (#19088)
[CI/Build] Remove V0 LoRA test (#19066)
Fix underscores in dict keys passed via CLI (#19030)
[Bugfix] disable processor cache (#19068)
[Doc] Improve the Pull Request template with key components (#19086)
[Misc] Add missing _Backend enums (#19081)
[Misc] fix: add miss best_of param validation (#18555)
[Misc] Add SPDX-FileCopyrightText (#19100)
[Doc] Readme standardization (#18695)
[doc] update docker version (#19074)
[Kernel] DeepEP dispatch-combine kernel integration (#18434)
[V1] Support cross-layer KV sharing (#18212)
[Perf] Tune scaled_fp8_quant by increasing vectorization (#18844)
Fix interaction between Optional and Annotated in CLI typing (#19093)
[v1] Re-init input batch for multiple kv cache groups (#18654)
[V1][Spec Decode][Ngram] 1.35x gain -> 1.95x gain on InstructCoder with prompt fix (#18971)
[Bugfix] get_num_blocks_to_allocate with null_block (#19031)
[Bugfix]: Fix the incompatibility issue with tool_choice 'required' when Thinking is enabled (#19075)
[Bugfix][P/D] Fix Prefix Cache Bug (#18411)
[Bugfix] Max concurrency estimation and check_enough_kv_cache_memory for models with sliding window layers (#19029)
feat: add data parallel rank to KVEventBatch (#18925)
[Misc] Fix path and python alias errors in disagg_prefill exmaples (#18919)
[Docs] Add developer doc about CI failures (#18782)
[CPU] V1 support for the CPU backend (#16441)
[Core] Cast multimodal input in hf processor (#18862)
[KERNEL] Sampler. CUDA kernel for applying repetition penalty (#18437)
[Cleanup][v1]:remote guided-decoding-backend for example (#19059)
[NVIDIA] Add Cutlass MLA backend (#17625)
[Bugfix] Fix FA3 full cuda graph correctness (#19106)
Fix #19130 (#19132)
[TPU] Skip hanging tests (#19115)
Fix ValueError: Missing value for tag key(s): model_name,engine. (#19113)
[Misc] Add packages for benchmark as extra dependency (#19089)
Improve the output precision of embedding models (#19092)
[CI/Build][Bugfix] Ensure compatibility with transformers 4.52 (#18678)
Add DeepSeek-R1-0528 function call chat template (#18874)
Sm100 blockwise fp8 swap ab (#18564)
[Doc] Update V1 Guide for embedding models (#19141)
Allow AsyncLLMEngine.generate to target a specific DP rank (#19102)
[Bugfix][EP+DP] Fix internode check (#19112)
[Perf] Tunings for SM100 FP8 CUTLASS kernel (#18778)
[TPU] Update dynamo dump file name in compilation test (#19108)
[Bugfix] fix v1 cpu worker fails on macOS (#19121)
[Kernel] Integrate batched/masked deepgemm kernel (#19111)
[Misc] refactor: simplify EngineCoreClient.make_async_mp_client in AsyncLLM (#18817)
[P/D] Heterogeneous TP (#18833)
[doc] small fix (#19167)
[Bugfix][Nixl] Fix full prefix cache hit bug (#18632)
[Bugfix] Fix port handling in make_zmq_path (#19117)
[Torch Nightly]add missing dependency (#18770)
Handle non-serializable objects when dumping benchmark results (#19114)
[BugFix][Minor] Fix full cuda graph bug when max_num_seqs < 512 (#19171)
[Bugfix]: Fix the incompatibility issue with stream when Thinking is disabled (#19135)
[Build] Annotate wheel and container path for release workflow (#19162)
[Misc] Remove unnecessary fallback to prefill-decode attention (#19138)
[Misc] Do not override NCCL_CUMEM_ENABLE if set explicitly (#19105)
[Frontend] improve vllm run-batch --help display (#19187)
[Bugfix] properly catch PIL-related errors for vision models when incorrect data urls are provided (#19202)
[mistral_common] Add v11 tokenizer (#19193)
Add H20-3e fused MoE kernel tuning configs for DeepSeek-R1/V3 (#19205)
[Hardware][NVIDIA] FP4 MoE kernel optimization (#19110)
[MISC][Bugfix] Use less CPU when message queue has been empty for some time (#16226)
[P/D][NixlConnector] Enable FlashInfer backend (#19090)
[Quantization] Skip Fp4 Test for compressed-tensors (#19217)
[V1] Use FlashInfer by default on Blackwell GPUs (#19118)
[Model] NemotronH support (#18863)
Fix AOPerModuleConfig name changes (#18869)
[Bugfix] Fix EAGLE vocab embedding construction for Llama 70B (#19033)
[v1] Hybrid Memory Allocator (#17996)
[TPU] update torch_xla pin (#19231)
Support allowed_token_ids in ChatCompletionRequest (#19143)
[Chore] update CODEOWNERS (#19247)
[v1][P/D] Fix a edge case in kv cache schedule (#19182)
[TPU] fix kv cache dtype in model runner (#19244)
[Quantization] Bump compressed-tensors version; update NVFP4A16 test model (#19224)
[Docs] Improve V1 KVConnector interface documentation (#19172)
Fix CompilationConfig repr (#19091)
Unit Test for run_dp_sharded_vision_model (#19103)
[Model] Optimize nemotron_h implementation (#19249)
[Core] Raise when non-multi-instance DP clients target a DP rank (#19227)
improve logits bias (#19041)
Fixed ppc build when it runs on non-RHEL based linux distros (#18422)
[BugFix] Fix MultiConnector test after HMA changes (#19291)
[Bugfix][Core] Update cancellation logic in generate() to handle Generator exits (#19225)
[Core] Fix abrupt request abort (#18485)
[BugFix] Fix tpu_model_runner block_id concatenation (#19228)
[Misc][Tools][Benchmark] Fix and improve auto tune script (#19163)
[Build][ROCm] Update Dockerfile.rocm (#19296)
[Easy][Test] Simplify test_function_tool_use with multiple parametrizes (#19269)
[Kernel] Integrate CUTLASS MoE kernel with PPLX (#18762)
[TPU][Test] Add script to run benchmark on TPU for buildkite (#19039)
[CI][PowerPC] Use a more appropriate way to select testcase in tests/models/language/pooling/test_embedding.py (#19253)
Add FlexAttention to V1 (#16078)
[Misc] refactor context extension (#19246)
[CI/Build] Improve Llama GGUF test robustness (#19287)
[Nit][Benchmark]Fix example in benchmark_serving_structured_output.py (#19311)
[AMD] Update compatible packaging version (#19309)
[BugFix][V1] Fix memory profiling bug (#18974)
[Bugfix]: Fix TypeError: 'float' object cannot be interpreted as an integer (#19283)
[Bugfix] Re-enable use_cudagraph in vLLM v1 (#19299)
[Misc] Change tests/compile to use VLLM_V1 by default (#19302)
Add H20-3e fused MoE kernel tuning configs for Qwen3-235B-A22B (#19315)
[Hardware][POWER] Add IBM POWER11 Support to CPU Extension Detection (#19082)
[Quantization] Add compressed-tensors NVFP4 support (#18312)
[Multi Modal] Add an env var for message queue max chunk bytes (#19242)
[Bugfix] model_max_length should consider max_model_len in tokenizer_config (#19201)
[Deprecation] Remove inputs arg fallback in Engine classes (#18799)
[Misc] Add documentation update reminder to PR template (#19289)
[Frontend] Remove unreachable code from llm.py (#19288)
[Misc] Cleanup compilation tests (#19343)
[doc] improve ci doc (#19307)
[Doc] Fix description in the Automatic Prefix Caching design doc (#19333)
[CI/Build] Fix LoRA test (#19350)
[Fix] Allow kernel compilation for CUDA capability 8.7 (#19328)
[CI] Introduce rules for llama auto-label (#19323)
[Docs] Fix a bullet list in usage/security.md (#19358)
[full_graph] Fix query_start_loc padding (#19321)
[v1] Add fp32 support to v1 engine through flex attn (#19319)
[Misc] Fixes and Optimizations for DeepEP + DeepGEMM combination. (#19298)
[Bugfix][Core] Prevent token lengths exceeding max_model_len in V0 (#19348)
[Quantization] Bump compressed-tensors version (#19295)
[Frontend] Make TIMEOUT_KEEP_ALIVE configurable through env var (#18472)
[TPU]Fix KV cache sharing tests (#19371)
[HOT-FIX] Add kv_sharing_target_layer_name argument to cutlass_mla backend (#19374)
[Misc] Fix a config typo in disable_hybrid_kv_cache_manager configuration (#19383)
[V1] Reuse V0's memory_profiling util for gpu worker memory profiling (#19312)
[Bugfix] Fix benchmark_moe.py (#19016)
Use xla flag to improve the quantized model performance (#19303)
Fix docs/mkdocs/hooks/remove_announcement.py (#19382)
[Frontend] Add tqdm_leave_pbar to control progress bar visibility (#19357)
[Core] Use tuple for kv cache group block ids (#19175)
[Bugfix] Fix modelscope token passed in (#19389)
[Core] Batch multi modal input using pinned memory (#19169)
Add security warning to bug report template (#19365)
[Misc] refactor neuron_multimodal and profiling (#19397)
Add clear documentation around the impact of debugging flag (#19369)
Automatically bind CPU OMP Threads of a rank to CPU ids of a NUMA node. (#17930)
Revert "[v1] Add fp32 support to v1 engine through flex attn" (#19404)
[BugFix][FlashInfer] Fix attention backend interface mismatch with unexpected keyword use_irope (#19134)
[BugFix][CPU] Fix CPU CI by ignore collecting test_pixtral (#19411)
Simplify ep kernels installation (#19412)
[Misc] Slight improvement of the BNB (#19418)
fix
fix
update config
add

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

Test Plan

Test Result

(Optional) Documentation Update

github-actions · 2025-07-08T17:09:04Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

tanujtiwari1998 requested a review from simon-mo as a code owner July 8, 2025 17:08

tanujtiwari1998 merged commit 99c5593 into main Jul 8, 2025
4 of 6 checks passed

tanujtiwari1998 force-pushed the main branch from a992216 to 6369426 Compare July 8, 2025 18:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

cached tokens completions #22

cached tokens completions #22

Uh oh!

tanujtiwari1998 commented Jul 8, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

Uh oh!

Uh oh!

cached tokens completions #22

cached tokens completions #22

Uh oh!

Conversation

tanujtiwari1998 commented Jul 8, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Essential Elements of an Effective PR Description Checklist

Purpose

Test Plan

Test Result

(Optional) Documentation Update

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

Uh oh!

Uh oh!

tanujtiwari1998 commented Jul 8, 2025 •

edited by github-actions bot

Loading