Update PyTorch to 2.7.0 #16859

huydhn · 2025-04-18T16:55:05Z

Notable changes:

PyTorch 2.7.0 has dropped CUDA 12.4, so the remaining options are 12.6 and 12.8
- CUDA 12.6 has build issue [Installation]: Fail to build vLLM from source on CUDA 12.6 #15435 (comment), so only 12.8 remains
We need a new ~~xformers~~ (0.0.30 is ready now), flashinfer, and mamba-ssm packages, so let build them from source for now. They can be installed from pypi once they are built upstream with 2.7.0
Leave XPU for later for Intel folks to pick it up as it requires a newer version of intel-extension-for-pytorch
Update the release pipeline to build with CUDA 12.8 too

Signed-off-by: Huy Do <huydhn@gmail.com>

github-actions · 2025-04-18T16:55:16Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

mergify · 2025-04-18T16:55:44Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @huydhn.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Huy Do <huydhn@gmail.com>

This reverts commit 1be359a. Signed-off-by: Huy Do <huydhn@gmail.com>

Signed-off-by: Huy Do <huydhn@gmail.com>

mgoin · 2025-04-23T17:17:13Z

Now that torch 2.7 has released (https://pypi.org/project/torch/2.7.0/) can this be updated?

huydhn · 2025-04-23T17:26:41Z

Now that torch 2.7 has released (https://pypi.org/project/torch/2.7.0/) can this be updated?

Yup, it can be updated now. Let me start working on that. On the other hand, I think I will keep the state of this PR as a reference because we plan to do similar validation for the next PyTorch release. Ideally, the validation needs to be done with PyTorch RC before publishing to pypi.

drisspg · 2025-04-23T17:27:32Z

docker/Dockerfile

+
+# TESTING: install xformers from source to test 2.7.0 final RC
+RUN --mount=type=cache,target=/root/.cache/uv \
+    TORCH_CUDA_ARCH_LIST='7.5 8.0 8.6 8.9 9.0+PTX' \


for blackwell support we will wnat to add sm100 here, although the +PTX should handle this..

Yes 10.0 as well as 12.0. +PTX is not enough. (Same comment in the several places this sequence appears.) cc @kushanam

As the size of this PR is relative big already and its signals are ready, let me add this change to a subsequent PR. It would be clearer that way I guess

drisspg · 2025-04-23T17:28:07Z

docker/Dockerfile

+    --index-strategy unsafe-best-match
+
+# TESTING: install xformers from source to test 2.7.0 final RC
+RUN --mount=type=cache,target=/root/.cache/uv \


If the intention is that the we are just going to wait till xformers releases a 12.8 compat whl though ignore above

IMO, I think we can just build the package from source for CI, then switch to the official xformers package once it's ready

How long does this take to compile xformers? If it's too long I don't want to slow down our CI time for this.

Let me dig out the number for this once the build finish. Without caching, it would be significant from what I see locally, but let's see what it takes to build once this is cached.

DogeFlow · 2025-04-30T03:43:50Z

When to sync to pip repository

vadimkantorov · 2025-05-04T19:19:44Z

Does vllm publish nightlies to some pip channel? Asking to try out vllm with pytorch 2.7.0

mgoin · 2025-05-04T19:25:01Z

@vadimkantorov

pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly

vadimkantorov · 2025-05-05T06:36:56Z

Couldn't find this nightly instruction in the installation section of README. Might be good to add it there too!

simon-mo · 2025-05-08T17:54:31Z

https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html#pre-built-wheels

vadimkantorov · 2025-05-12T10:59:45Z

pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly

For me, this again tries to fetch 2.6.0 :( despite having 2.7.0 installed:

vllm-0.8.5.post1-cp38-abi3-manylinux1_x86_64.whl does not look like nightly wheel :( looks like it's the release wheel. I think pip discovers this wheel from my local cache because it exists and does not attempt to install the nightly :( Bug in pip? I manually went to https://wheels.vllm.ai/nightly/vllm and found this wheel https://wheels.vllm.ai/vllm-0.8.5.dev599%2Bg9fbf2bfbd-cp38-abi3-manylinux1_x86_64.whl - which indeed looks like nightly, but it gives 404 :(

$ pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
                                                                                                          
Defaulting to user installation because normal site-packages is not writeable                                              
Looking in indexes: https://pypi.org/simple, https://wheels.vllm.ai/nightly                                                
Collecting vllm                                                                                                              
Downloading vllm-0.8.5.post1-cp38-abi3-manylinux1_x86_64.whl (326.4 MB)                                                        
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 326.4/326.4 MB 10.2 MB/s eta 0:00:00
...
Collecting torch==2.6.0
  Downloading torch-2.6.0-cp310-cp310-manylinux1_x86_64.whl (766.7 MB)
     ━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 221.3/766.7 MB 252.7 MB/s eta 0:00:03                                        
ERROR: Operation cancelled by user

# second try:
$ pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/vllm                                                                                                          
Defaulting to user installation because normal site-packages is not writeable                                              
Looking in indexes: https://pypi.org/simple, https://wheels.vllm.ai/nightly/vllm                                           
Collecting vllm
  Using cached vllm-0.8.5.post1-cp38-abi3-manylinux1_x86_64.whl (326.4 MB)

Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>

vadimkantorov · 2025-05-12T11:33:23Z

Basically, can't find direct URLs to nightly wheels :( which might be needed to circumvent pip not wanting to install the nightly for some reason

So far managed to find published commit from:

pip index versions vllm --pre --extra-index-url https://wheels.vllm.ai/nightly

# and then

pip install -U vllm==0.8.5.dev600+g7ea6cb28b --pre --extra-index-url https://wheels.vllm.ai/nightly

Signed-off-by: Yuqi Zhang <yuqizhang@google.com>

This PR fixes the other issue discovered in vllm-project#16859 when upgrading from PyTorch 2.6 to PyTorch 2.7. I don't know why the code used to work in PyTorch 2.6, but the explanation is: - when we are running PiecewiseCompileInterpreter, we end up doing FakeTensor propagation - FakeTensor propagation requires `enable_python_dispatcher` to work. The mechanism is that some of our "C++ implementations" for operations, like matmul, force specialization of dynamic shapes. torch.compile works around this by replacing PyTorch's "C++ implementation" for matmul with a python-based implementation for matmul that does not force specialization. Test Plan: - Ran `pytest -v tests/models/test_transformers.py -k test_models[meta-llama/Llama-3.2-1B-Instruct-transformers]` with PyTorch >= 2.7 and vllm-project#17330, verified that the test passes. Signed-off-by: rzou <zou3519@gmail.com>

zhanglianjie-163 · 2025-06-10T03:02:32Z

Hi，why upgrade to pytorch2.7?

Signed-off-by: minpeter <kali2005611@gmail.com>

jessiewiswjc · 2025-06-25T03:39:07Z

I found that after this commit, the tpot/itl of the Qwen/Qwen2.5-14B-Instruct model on an h20 dropped from 20ms to 10ms. I want to know which part of the code benefited from this. Is it pytorch2.7?

Update to PyTorch 2.7.0

3498754

Signed-off-by: Huy Do <huydhn@gmail.com>

huydhn requested a review from tlrmchlsmth as a code owner April 18, 2025 16:55

mergify bot added the ci/build label Apr 18, 2025

mergify bot added the needs-rebase label Apr 18, 2025

Merge branch 'main' into torch-2.7.0-final-rc

a131606

Signed-off-by: Huy Do <huydhn@gmail.com>

mergify bot removed the needs-rebase label Apr 18, 2025

mgoin self-requested a review April 18, 2025 17:07

huydhn added 17 commits April 18, 2025 12:43

Install xformers from source

52a7c4b

Signed-off-by: Huy Do <huydhn@gmail.com>

Wrong installation order

c33698e

Signed-off-by: Huy Do <huydhn@gmail.com>

Use --index-strategy unsafe-best-match temporarily

363303b

Signed-off-by: Huy Do <huydhn@gmail.com>

More tweak

174c2a1

Signed-off-by: Huy Do <huydhn@gmail.com>

Try CUDA 12.6 then

1be359a

Signed-off-by: Huy Do <huydhn@gmail.com>

Revert "Try CUDA 12.6 then"

32fe6df

This reverts commit 1be359a. Signed-off-by: Huy Do <huydhn@gmail.com>

More tweak for CPU, XPU and ROCm

a0e5706

Signed-off-by: Huy Do <huydhn@gmail.com>

Leave XPU for now

3b27235

Signed-off-by: Huy Do <huydhn@gmail.com>

Merge branch 'main' into torch-2.7.0-final-rc

100b6a7

Use 2.7.0 final RC for now till it's ready on pypi

079c59b

Signed-off-by: Huy Do <huydhn@gmail.com>

Fix a minor import bug from XPU

99b7173

Signed-off-by: Huy Do <huydhn@gmail.com>

Another --index-strategy unsafe-best-match

1f62dc1

Signed-off-by: Huy Do <huydhn@gmail.com>

Install xformers and flashinfer from source

2b70bf5

Signed-off-by: Huy Do <huydhn@gmail.com>

Add --no-build-isolation

6e473d8

Signed-off-by: Huy Do <huydhn@gmail.com>

Install xformers on vllm-base

72f27d0

Signed-off-by: Huy Do <huydhn@gmail.com>

Keep xformers in cuda.txt and set TORCH_CUDA_ARCH_LIST

86a6954

Signed-off-by: Huy Do <huydhn@gmail.com>

Build xformers separately because it depends on torch (to build?)

71ca7fc

Signed-off-by: Huy Do <huydhn@gmail.com>

drisspg reviewed Apr 23, 2025

View reviewed changes

simon-mo approved these changes Apr 30, 2025

View reviewed changes

simon-mo merged commit 2c4f59a into vllm-project:main Apr 30, 2025
87 of 90 checks passed

mgoin mentioned this pull request Apr 30, 2025

Support torch==2.7.0 LMCache/LMCache#566

Closed

noooop mentioned this pull request May 1, 2025

[CI] Add mteb testing to test the accuracy of the embedding model #17175

Merged

22quinn mentioned this pull request May 2, 2025

Update test requirements to CUDA 12.8 #17576

Merged

radeksm pushed a commit to radeksm/vllm that referenced this pull request May 2, 2025

Update PyTorch to 2.7.0 (vllm-project#16859)

4d2b2f3

hickeyma mentioned this pull request May 6, 2025

fix: Pin precise torch version to avoid vLLM compatability issues LMCache/LMCache#583

Merged

vadimkantorov mentioned this pull request May 12, 2025

[Usage]: Compat with vllm 0.8.5 or even nightly (which supports of PyTorch 2.7.0) and support for PyTorch 2.7.0 #17987

Closed

1 task

RichardoMrMu pushed a commit to RichardoMrMu/vllm that referenced this pull request May 12, 2025

Update PyTorch to 2.7.0 (vllm-project#16859)

6fa7a6f

Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>

yankay mentioned this pull request May 22, 2025

[Hardware][CPU] Update intel_extension_for_pytorch 2.7.0 and move to requirements/cpu.txt #18542

Merged

zzzyq pushed a commit to zzzyq/vllm that referenced this pull request May 24, 2025

Update PyTorch to 2.7.0 (vllm-project#16859)

2403dd8

Signed-off-by: Yuqi Zhang <yuqizhang@google.com>

huydhn mentioned this pull request Jun 16, 2025

Add a doc on how to update PyTorch version #19705

Merged

minpeter pushed a commit to minpeter/vllm that referenced this pull request Jun 24, 2025

Update PyTorch to 2.7.0 (vllm-project#16859)

62254f7

Signed-off-by: minpeter <kali2005611@gmail.com>

Uh oh!

Update PyTorch to 2.7.0 #16859

Update PyTorch to 2.7.0 #16859

Conversation

huydhn commented Apr 18, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 18, 2025

Uh oh!

mergify bot commented Apr 18, 2025

Uh oh!

mgoin commented Apr 23, 2025

Uh oh!

huydhn commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drisspg Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

cliffwoolley Apr 30, 2025

Choose a reason for hiding this comment

Uh oh!

huydhn Apr 30, 2025

Choose a reason for hiding this comment

Uh oh!

drisspg Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

huydhn Apr 25, 2025

Choose a reason for hiding this comment

Uh oh!

simon-mo Apr 26, 2025

Choose a reason for hiding this comment

Uh oh!

huydhn Apr 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

DogeFlow commented Apr 30, 2025

Uh oh!

vadimkantorov commented May 4, 2025

Uh oh!

mgoin commented May 4, 2025

Uh oh!

vadimkantorov commented May 5, 2025

Uh oh!

simon-mo commented May 8, 2025

Uh oh!

vadimkantorov commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vadimkantorov commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhanglianjie-163 commented Jun 10, 2025

Uh oh!

jessiewiswjc commented Jun 25, 2025

Uh oh!

Uh oh!

huydhn commented Apr 18, 2025 •

edited by github-actions bot

Loading

huydhn commented Apr 23, 2025 •

edited

Loading

huydhn Apr 26, 2025 •

edited

Loading

vadimkantorov commented May 12, 2025 •

edited

Loading

vadimkantorov commented May 12, 2025 •

edited

Loading