[paddle-TRT]support matmul set to int8 in multihead #34917
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
ceci3
changed the title
ceci3
requested review from
cryoco,
qingqing01,
shangzhizhou and
wanghaoshuang
ceci3
changed the title
| BOOST_GET_CONST(float, add1_op_desc->GetAttr("out_threshold")); | ||
| auto out_scale2 = | ||
| BOOST_GET_CONST(float, mul2_op_desc->GetAttr("out_threshold")); | ||
| BOOST_GET_CONST(float, add2_op_desc->GetAttr("out_threshold")); |
There was a problem hiding this comment.
out_scale0，out_scale1 ，out_scale2分别是q,k,v的scale，将其最大值设置为 multihead_op的"out_threshold"， multihead_op中包含FC和QKVToContextPlugin，这个"out_threhold"是用作FC的output scale.
问题是：能給"out_threhold"换个名字么？原因如下：
- 当前命名不太直观；容易被误以为multihead_op的output scale;
- "out_threhold"用于fusion pass向TRT op convert传递信息，所以这个命名不用受限于量化模型的存储格式。另外，也不受OP Maker中对attr的约束。
| multihead_op_desc.SetAttr("dp_probs", qkv_plugin_scale); | ||
| } | ||
| auto* matmul_qk_op_desc = matmul_qk->Op(); | ||
| if (matmul_qk_op_desc->HasAttr("X_scale")) { |
There was a problem hiding this comment.
如果这个判断为False，line897~902还有必要么？
|
|
||
| float* weight_data = nullptr; | ||
| bool enable_int8 = op_desc.HasAttr("enable_int8"); | ||
| bool qkv_plugin_int8 = |
There was a problem hiding this comment.
命名更直观些？比如包含关键字"qkv2context".
| dp_probs = out_scale / 127.0; | ||
| dp_probs = | ||
| BOOST_GET_CONST(float, op_desc.GetAttr("dp_probs")) / 127.0; | ||
| dp_probs = dp_probs / 127.0; |
Comment on lines
169
to
176
| ? nvinfer1::DataType::kHALF | ||
| : nvinfer1::DataType::kFLOAT); | ||
| if (enable_int8) { | ||
| type = static_cast<int>(nvinfer1::DataType::kHALF); | ||
| if (qkv_plugin_int8) { | ||
| type = static_cast<int>(nvinfer1::DataType::kINT8); | ||
| } else { | ||
| type = static_cast<int>(nvinfer1::DataType::kHALF); | ||
| } |
There was a problem hiding this comment.
确定这些条件(line169-line176)判断没有冗余么？比如line175可以删除么？
wanghaoshuang
approved these changes
Aug 30, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Performance optimization
PR changes
Others
Describe
matmul can be set to int8 in multihead
envs: cuda10.2, cudnn 8.0 trt: 7.2.2.3
int8计算结果会因为trt底层kernel选择不同结果会有些浮动。