Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[torch.compile] directly register custom op #9896

Merged
merged 25 commits into from
Nov 1, 2024
Merged

[torch.compile] directly register custom op #9896

merged 25 commits into from
Nov 1, 2024

Conversation

youkaichao
Copy link
Member

see https://gist.github.com/youkaichao/ecbea9ec9fc79a45d2adce1784d7a9a5 for the discussion on the perf implification of registering custom ops

@youkaichao
fix register of fused moe
29e77d9 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
fix marlin
ba4f3ad 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
parallel state
19b4bff 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
fix parallel state
a09f3cb 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
add toy llama
12cac28 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
add test simple
7f0fedb 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
flash attn
15c7728 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
flashinfer
e3ca1b6 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
v1.flash_attn
f7e2966 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

@youkaichao
Copy link
Member Author

the schema is generated by using torch.library.custom_op

@youkaichao
utils
f8fe3dd 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
flashinfer
8334099 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
simple
5ec48b9 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
toy llama
449121c 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
flash attn
eca0f28 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
parallel state
dc23b25 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
marlin
cb4f157 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
fused moe
3c5e7dd 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
v1.flash_attn
0547328 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
add comments
d87bfe0 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
Copy link
Member Author

need to solve pytorch/pytorch#139444 first

@youkaichao
hack fix library
d515d61 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
fix doc build
012533a 
Signed-off-by: youkaichao <youkaichao@gmail.com>
tlrmchlsmth
tlrmchlsmth reviewed Nov 1, 2024
View reviewed changes
vllm/utils.py Outdated Show resolved Hide resolved
@youkaichao
fix lifetime
2ef5e40 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
add lib arg
65945e0 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
fix
529b28a 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
fix doc build
d4b1399 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao youkaichao added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 1, 2024
@youkaichao youkaichao merged commit 96e0c9c into vllm-project:main Nov 1, 2024
34 of 49 checks passed
@youkaichao youkaichao deleted the register branch November 1, 2024 04:56
@youkaichao youkaichao mentioned this pull request Nov 2, 2024
Merged
lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Nov 4, 2024
@youkaichao
[torch.compile] directly register custom op (vllm-project#9896)
cbdb2c6 
Signed-off-by: youkaichao <youkaichao@gmail.com>
lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Nov 4, 2024
@youkaichao
[torch.compile] directly register custom op (vllm-project#9896)
d547ee8 
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Linkun Chen <github+anyscale@lkchen.net>
richardsliu pushed a commit to richardsliu/vllm that referenced this pull request Nov 4, 2024
@youkaichao @richardsliu
[torch.compile] directly register custom op (vllm-project#9896)
0458bc6 
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Richard Liu <ricliu@google.com>
bigPYJ1151 pushed a commit to bigPYJ1151/vllm that referenced this pull request Nov 5, 2024
@youkaichao @bigPYJ1151
[torch.compile] directly register custom op (vllm-project#9896)
c5a966c 
Signed-off-by: youkaichao <youkaichao@gmail.com>
hissu-hyvarinen pushed a commit to ROCm/vllm that referenced this pull request Nov 6, 2024
@youkaichao @hissu-hyvarinen
[torch.compile] directly register custom op (vllm-project#9896)
0eab1f9 
Signed-off-by: youkaichao <youkaichao@gmail.com>
JC1DA pushed a commit to JC1DA/vllm that referenced this pull request Nov 11, 2024
@youkaichao @JC1DA
[torch.compile] directly register custom op (vllm-project#9896)
2151ed5 
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Loc Huynh <jc1da.3011@gmail.com>
sumitd2 pushed a commit to sumitd2/vllm that referenced this pull request Nov 14, 2024
@youkaichao @sumitd2
[torch.compile] directly register custom op (vllm-project#9896)
a9c1021 
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Sumit Dubey <sumit.dubey2@ibm.com>
KuntaiDu pushed a commit to KuntaiDu/vllm that referenced this pull request Nov 20, 2024
@youkaichao
[torch.compile] directly register custom op (vllm-project#9896)
5e88700 
Signed-off-by: youkaichao <youkaichao@gmail.com>
mfournioux pushed a commit to mfournioux/vllm that referenced this pull request Nov 20, 2024
@youkaichao @mfournioux
[torch.compile] directly register custom op (vllm-project#9896)
c34523c 
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>
@jeejeelee jeejeelee mentioned this pull request Nov 21, 2024
Merged
tlrmchlsmth pushed a commit to neuralmagic/vllm that referenced this pull request Nov 23, 2024
@youkaichao @tlrmchlsmth
[torch.compile] directly register custom op (vllm-project#9896)
27bf057 
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request Dec 13, 2024
@youkaichao
[torch.compile] directly register custom op (vllm-project#9896)
c875585 
Signed-off-by: youkaichao <youkaichao@gmail.com>
sergeykochetkov pushed a commit to sergeykochetkov/vllm_spec_decoding that referenced this pull request Dec 27, 2024
@youkaichao
[torch.compile] directly register custom op (vllm-project#9896)
dd3a2ae 
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: s.kochetkov <s.m.kochetkov@tcsbank.ru>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tlrmchlsmth tlrmchlsmth tlrmchlsmth approved these changes

Assignees
No one assigned
Labels
ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@youkaichao @tlrmchlsmth