support aclgraph #426
Merged
support aclgraph #426
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
documentation
github-actions
bot
removed
documentation
wuhuikx
reviewed
Apr 14, 2025
fakeYan
commented
Apr 15, 2025
fakeYan
force-pushed
the
main
branch
7 times, most recently
from
1ae4054 to
695689e
Compare
fakeYan
force-pushed
the
main
branch
2 times, most recently
from
8793fa4 to
edce3b8
Compare
wangxiyuan
reviewed
Apr 18, 2025
| @@ -171,6 +179,12 @@ def __init__(self, vllm_config: VllmConfig, device: torch.device): | |||
| self.input_positions_cpu = torch.arange(0, | |||
| self.max_num_tokens, | |||
| device="cpu") | |||
| self.use_cuda_graph = (self.vllm_config.compilation_config.level | |||
There was a problem hiding this comment.
rename to self.use_acl_graph
There was a problem hiding this comment.
self.use_npu_graph is better
| self.use_cuda_graph = (self.vllm_config.compilation_config.level | ||
| == CompilationLevel.PIECEWISE | ||
| and not self.model_config.enforce_eager) | ||
| self.cudagraph_batch_sizes = list( |
| from vllm.v1.sample.rejection_sampler import INVALID_TOKEN_ID, RejectionSampler | ||
| else: | ||
| INVALID_TOKEN_ID = None | ||
| RejectionSampler = None |
There was a problem hiding this comment.
why this change? HAS_TRITON is alway false in vllm-ascend. So I guess you want to rewrite vllm.v1.sample.rejection_sampler. INVALID_TOKEN_ID, RejectionSampler in vllm-ascend here？
vllm_ascend/utils.py
Outdated
| self.name = name | ||
|
|
||
|
|
||
| def register_dummy_fusion_op() -> None: |
requirements.txt
Outdated
| torch >= 2.5.1 | ||
| torch_npu == 2.5.1rc1 |
There was a problem hiding this comment.
do not limit torch-npu version here.
fakeYan
force-pushed
the
main
branch
2 times, most recently
from
e6bdffb to
a8d3d27
Compare
yiz-liu
force-pushed
the
main
branch
8 times, most recently
from
28ce6ee to
0ca634e
Compare
MengqingCao
reviewed
Apr 22, 2025
vllm_ascend/__init__.py
Outdated
| @@ -15,6 +15,8 @@ | |||
| # This file is a part of the vllm-ascend project. | |||
| # | |||
|
|
|||
| from torch_npu.contrib import transfer_to_npu # noqa: F401 | |||
There was a problem hiding this comment.
Why do we need this here? This will hide some issue and break some scenes in RL, where torch.cuda expected to be called normally.
yiz-liu
force-pushed
the
main
branch
6 times, most recently
from
eca5828 to
6c0e10d
Compare
Signed-off-by: Bug Hunter Yan <yanpq@zju.edu.cn>
yiz-liu
force-pushed
the
main
branch
2 times, most recently
from
a7e0e28 to
b2a0b53
Compare
Signed-off-by: Yizhou Liu <liu_yizhou@outlook.com>
ganyi1996ppo
approved these changes
Apr 23, 2025
ttanzhiqiang
pushed a commit
to ttanzhiqiang/vllm-ascend
that referenced
this pull request
Apr 27, 2025
<!-- Thanks for sending a pull request! BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing/overview.html --> ### What this PR does / why we need it? <!-- - Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. If possible, please consider writing useful notes for better and faster reviews in your PR. - Please clarify why the changes are needed. For instance, the use case and bug description. - Fixes # --> This PR supports the access of vllm-acend to the piecewise_graph feature provided by the v1 engine. 1. register unifiled_ascend_attention_with_output for piecewise_graph to split graph. 2. support NPUGraph to accelerate kernel launch. ### Does this PR introduce _any_ user-facing change? <!-- Note that it means *any* user-facing change including all aspects such as API, interface or other behavior changes. Documentation-only updates are not considered user-facing changes. --> support npugraph to default， Users can disenable the npugraph feature by configuring enforce_eager. This has corresponding requirements for the versions of torch_npu and CANN, and they need to support graph capture. ### How was this patch tested? <!-- CI passed with new added/existing test. If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future. If tests were not added, please describe why they were not added and/or why it was difficult to add. --> it turn to default --------- Signed-off-by: Bug Hunter Yan <yanpq@zju.edu.cn> Signed-off-by: Yizhou Liu <liu_yizhou@outlook.com> Co-authored-by: Yizhou Liu <liu_yizhou@outlook.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it?
This PR supports the access of vllm-acend to the piecewise_graph feature provided by the v1 engine.
Does this PR introduce any user-facing change?
support npugraph to default， Users can disenable the npugraph feature by configuring enforce_eager，just like.
from vllm import LLM, llm = LLM(model="Qwen/Qwen2.5-0.5B-Instruct", enforce_eager=True)This has corresponding requirements for the versions of torch_npu and CANN, and they need to support graph capture.
How was this patch tested?
it turn to default