[Paddle Inference] Add bias input of mmha and simplify mmha. #56411
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
你的PR提交成功，感谢你对开源项目的贡献! |
| out->set_dims(out_dims); | ||
| out->set_dtype(x.dtype()); | ||
| } | ||
|
|
There was a problem hiding this comment.
量化的完整需求是
输入为int32/float16/32
输出为int8/float/16/32
我看这里考虑了输出是int8的情况 但是没考虑输入是int32的情况是么
|
|
||
| namespace phi { | ||
| namespace fusion { | ||
|
|
There was a problem hiding this comment.
不需要添加多余的头文件，防止被其他调用
heavengate
reviewed
Aug 21, 2023
| Args: | ||
| x (Tensor): The input tensor could be 2-D tensor. Its shape is [batch_size, 3 * num_head * head_dim]. | ||
| cache_kvs (list(Tensor)|tuple(Tensor)): The cache structure tensors for the generation model. Its shape is [2, batch_size, num_head, max_seq_len, head_dim]. | ||
| bias (Tensor, optional): The bias tensor. Its shape is [3, num_head, head_dim]. |
There was a problem hiding this comment.
也需要加一下compte_dtype的参数说明~
| self.rotary_emb_dims = 0 | ||
| self.use_neox_rotary_style = False | ||
|
|
||
| self.compute_dtype = "default" |
There was a problem hiding this comment.
单测有int32的情况
zyfncg
approved these changes
Aug 22, 2023
jzhang533
approved these changes
Aug 22, 2023
| seq_len (int, optional): The seq_len, used to get input length. Default 1. | ||
| rotary_emb_dims (int, optional): The rotary_emb_dims. Default 1. | ||
| use_neox_rotary_style (bool, optional): A flag indicating whether neox_rotary_style is needed or not. Default False. | ||
| compute_dtype (string): A compute dtype, used to represent the input data type. |
There was a problem hiding this comment.
compute_dtype的为啥不能根据输入tensor的类型判断呢？
There was a problem hiding this comment.
ptq 情况下，输入 x 有可能是int32，如果根据 cache_kv dtype 判断，后续 cache_kv 量化支持还需要修改。
vivienfanghuagood
approved these changes
Aug 23, 2023
Aurelius84
approved these changes
Aug 23, 2023
lanxianghit
approved these changes
Aug 23, 2023
raindrops2sea
approved these changes
Aug 23, 2023
heavengate
approved these changes
Aug 23, 2023
BeingGod
pushed a commit
to BeingGod/Paddle
that referenced
this pull request
Sep 9, 2023
…addle#56411) * add_bias_and_simplify_mmha
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Others
PR changes
Others
Description
Add bias input of mmha and simplify mmha.
关联pr #55344
Pcard-71502