Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Platforms] Add device_type in Platform #10508

Merged
merged 1 commit into from
Nov 21, 2024
Merged

[Platforms] Add device_type in Platform #10508

merged 1 commit into from
Nov 21, 2024

Conversation

MengqingCao
Copy link
Contributor

@MengqingCao MengqingCao commented Nov 21, 2024 •
edited by github-actions bot
Loading

Part of #9268

@MengqingCao
[Platforms] add device_type in Platform
49a97a0 
Signed-off-by: MengqingCao <cmq0113@163.com>
@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

@MengqingCao
Copy link
Contributor Author

cc @youkaichao @DarkLight1337

DarkLight1337
DarkLight1337 approved these changes Nov 21, 2024
View reviewed changes
Copy link
Member

@DarkLight1337 DarkLight1337 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@DarkLight1337 DarkLight1337 enabled auto-merge (squash) November 21, 2024 02:55
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 21, 2024
@DarkLight1337 DarkLight1337 merged commit 9d82717 into vllm-project:main Nov 21, 2024
62 checks passed
youkaichao
youkaichao reviewed Nov 21, 2024
View reviewed changes
vllm/config.py
elif current_platform.is_xpu():
self.device_type = "xpu"
else:
self.device_type = current_platform.device_type
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will raise exception (attribute error) directly.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to set it in UnspecifiedPlatform too

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you tell me in what cases UnspecifiedPlatform would be called? I cannot confirm the device_type of that.

Copy link
Contributor Author

@MengqingCao MengqingCao Nov 21, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will raise exception (attribute error) directly.

UnspecifiedPlatform inherits from Platform, thus I think this won't happen.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we install a cpu version pytorch in gpu environment (like arm64 chips, GH200), i think this will happen.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to keep the original error message raise RuntimeError("Failed to infer device type") , but i'm afraid this will become an attribute after this pr.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your explanation, I got your point. Sorry for missing this case.

@youkaichao youkaichao mentioned this pull request Nov 21, 2024
Merged
tlrmchlsmth pushed a commit to neuralmagic/vllm that referenced this pull request Nov 23, 2024
@MengqingCao @tlrmchlsmth
[Platforms] Add device_type in Platform (vllm-project#10508)
f178e90 
Signed-off-by: MengqingCao <cmq0113@163.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
mfournioux pushed a commit to mfournioux/vllm that referenced this pull request Nov 28, 2024
@MengqingCao @mfournioux
[Platforms] Add device_type in Platform (vllm-project#10508)
552dea3 
Signed-off-by: MengqingCao <cmq0113@163.com>
Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>
sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request Dec 13, 2024
@MengqingCao
[Platforms] Add device_type in Platform (vllm-project#10508)
7b5cecf 
Signed-off-by: MengqingCao <cmq0113@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@youkaichao youkaichao youkaichao left review comments

@DarkLight1337 DarkLight1337 DarkLight1337 approved these changes

Assignees
No one assigned
Labels
ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@MengqingCao @youkaichao @DarkLight1337