[Core/Bugfix] Add FP8 K/V Scale and dtype conversion for prefix/prefill Triton Kernel #7208
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge). To run full CI, you can do one of these:
🚀 |
github-actions
bot
added
the
ready
|
@comaniac how can I trigger the CI? I have no dev env for vllm currently |
jon-chuang
changed the title
|
The CI is already triggered. |
|
I guess it is waiting for a runner to schedule |
|
Turns out I actually needed to push new commit after converting from draft status to trigger CI |
Does that mean you cannot verify this PR locally? We should avoid using CI to verify and debug because it's slow and costly. |
|
Hmm, actually; shouldn't Triton have support for mixed FP8 matmul (E5M2 only)?
EDIT:
|
…chuang/fix-fp8-triton-kernel
| E5M2_KV_MODELS = [ # type: ignore | ||
| # does not work with fp8 kv cache kernel | ||
| # - CUDA illegal memory access - undiagnosed | ||
| # "facebook/opt-125m", | ||
| ] |
There was a problem hiding this comment.
Does other models work with E5M2? For example does that work if you just use the FP16 Qwen2-1.5B-Instruct checkpoint?
There was a problem hiding this comment.
I can't seem to find any model that has quantized kv E5M2; if not explicitly quantized with KV scales, the log probs differ due to the inaccuracy during low-precision float conversion
There was a problem hiding this comment.
E5M2 doesn't need scaling factors and the accuracy shouldn't drop significantly. If it doesn't make sense to check the logprobs for E5M2, we could skip testing them.
There was a problem hiding this comment.
I see, the reason the test failed was because Qwen2-1.5B is actually BF16 (exponent - 8 bit).
I tried with llama-2-7b which is FP16 (exponent 5 bits) and the tests pass now.
| if chunked_prefill_token_size != 1: | ||
| enable_chunked_prefill = True | ||
| max_num_batched_tokens = chunked_prefill_token_size |
There was a problem hiding this comment.
So when chunked_prefill_token_size==1 you disable chunked prefill. Then isn't the following 2 runners always the same? In this case why we need to test this case (looks like the existing test case also has the same issue)?
There was a problem hiding this comment.
I'm not sure I was following the previous test blindly, which is totally wrong as you mentioned. If you want I can get rid of the parameter 1 for both.
Alternately I can also remove the if statement which will allow testing the code path with enable_chunked_prefill=True, prefill_size=1
There was a problem hiding this comment.
I remove the if statement from both which is useless as you said
There was a problem hiding this comment.
Yeah let's just make it reasonable. Also cc @rkooo567 who may know what it forms like this.
|
Thank you for help in reviewing @comaniac. Need your help to merge as I have no write access. |
|
@jon-chuang Is this to-do fixed with this MR? https://github.com/vllm-project/vllm/blame/main/vllm/attention/backends/xformers.py#L601 |
|
Yes, the comment should have been removed |
|
can #3234 be closed @jon-chuang ? |
|
Yes @chenxu2048 |
…ll Triton Kernel (vllm-project#7208) Co-authored-by: Cody Yu <hao.yu.cody@gmail.com> Signed-off-by: Alvant <alvasian@yandex.ru>
…ll Triton Kernel (vllm-project#7208) Co-authored-by: Cody Yu <hao.yu.cody@gmail.com> Signed-off-by: LeiWang1999 <leiwang1999@outlook.com>
Fix the FP8 Triton kernel issue. Should enable FP8 KV Cache to be used with:
FIX #4381 #3880 #3156 #3880
TODO:
Notes:
vllm/csrc/attention/attention_kernels.cu
Line 287 in a3bbbfa
vllm/csrc/attention/attention_kernels.cu
Line 417 in a3bbbfa
vllm/csrc/quantization/fp8/nvidia/quant_utils.cuh
Line 299 in a3bbbfa
Example Output (decode v.s. chunked prefill with FP8 KV Cache):
Performance is similar to decode in low load case on max_chunk=16,max_sequence_len=512
I attribute the slightly lower perf to lack of perf tuning of the triton kernels
PR Checklist (Click to Expand)
Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.
PR Title and Classification
Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:
[Bugfix]for bug fixes.[CI/Build]for build or continuous integration improvements.[Doc]for documentation fixes and improvements.[Model]for adding a new model or improving an existing model. Model name should appear in the title.[Frontend]For changes on the vLLM frontend (e.g., OpenAI API server,LLMclass, etc.)[Kernel]for changes affecting CUDA kernels or other compute kernels.[Core]for changes in the core vLLM logic (e.g.,LLMEngine,AsyncLLMEngine,Scheduler, etc.)[Hardware][Vendor]for hardware-specific changes. Vendor name should appear in the prefix (e.g.,[Hardware][AMD]).[Misc]for PRs that do not fit the above categories. Please use this sparingly.Note: If the PR spans more than one category, please include all relevant prefixes.
Code Quality
The PR need to meet the following code quality standards:
format.shto format your code.docs/source/if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.Notes for Large Changes
Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with
rfc-requiredand might not go through the PR.What to Expect for the Reviews
The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:
action-requiredlabel on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.Thank You
Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!
CC: @comaniac