Skip to content

Replace matmul(v2) with fused_matmul during oneDNN fuse passes #49515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 58 commits into from
Feb 3, 2023
Merged

Replace matmul(v2) with fused_matmul during oneDNN fuse passes #49515

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
8cf612a
replace matmul with matmul_v2 in fuse passes
Silv3S Dec 14, 2022
1a877be
Remove fusion logic from matmul
Silv3S Dec 14, 2022
71da01e
Merge branch 'develop' into clean_matmuls
Silv3S Dec 15, 2022
e47c071
removing fusion methods
Silv3S Dec 15, 2022
2d863cd
add proper name
Silv3S Dec 15, 2022
02fbcb9
adjust namespaces
Silv3S Dec 15, 2022
7df1a54
clean attrs in python tests
Silv3S Dec 21, 2022
c7b4e05
delete checkpoint and restore matmul version
Silv3S Dec 21, 2022
c98f126
remove unused code
Silv3S Dec 22, 2022
53f6ae5
Merge branch 'PaddlePaddle:develop' into remove_extra_matmul_attrs
Silv3S Dec 23, 2022
a301add
matmul and reshape/transpose fuses migrated
Silv3S Dec 23, 2022
3cff939
split MatmulOneDNN headers
Silv3S Dec 27, 2022
cc26096
fuse activation and eltwise_add
Silv3S Dec 27, 2022
a8c3774
add fuse_activation
Silv3S Dec 27, 2022
edb6615
matmul_transpose_reshape/reshape_transpose_matmul
Silv3S Dec 28, 2022
2bf6d5a
matmul + elementwise_add (fused)
Silv3S Dec 28, 2022
b0bf97e
activation temporary modifciation
Silv3S Dec 28, 2022
773f3c1
Merge branch 'develop' into remove_extra_matmul_attrs
Silv3S Jan 3, 2023
aee35ec
Merge branch 'PaddlePaddle:develop' into remove_extra_matmul_attrs
Silv3S Jan 3, 2023
cac16fb
merge newest develop
Silv3S Jan 4, 2023
1cfa75b
revert depedency from other PR
Silv3S Jan 4, 2023
b2e9baa
Merge branch 'remove_extra_matmul_attrs' of https://github.com/Silv3S…
Silv3S Jan 4, 2023
5eb838c
remove depedency from other PR
Silv3S Jan 4, 2023
dc3cf8e
revert pbtxt
Silv3S Jan 4, 2023
5833035
remove placeholders from matmul_v2
Silv3S Jan 4, 2023
5d71310
add description in OPMaker
Silv3S Jan 5, 2023
8195aa4
remove matmul_v2_op.h and all depedencies
Silv3S Jan 5, 2023
9e1c32b
remove dims changing in base op
Silv3S Jan 5, 2023
c7df785
add possibility to fuse already fused_matmul
Silv3S Jan 5, 2023
946bf04
Merge branch 'PaddlePaddle:develop' into remove_extra_matmul_attrs
Silv3S Jan 5, 2023
415202e
Merge branch 'PaddlePaddle:develop' into remove_extra_matmul_attrs
Silv3S Jan 8, 2023
52d43ca
Merge branch 'PaddlePaddle:develop' into remove_extra_matmul_attrs
Silv3S Jan 9, 2023
8687536
restart broken CI
Silv3S Jan 9, 2023
4bd883f
Empty-Commit
Silv3S Jan 9, 2023
35ad305
Merge branch 'PaddlePaddle:develop' into remove_extra_matmul_attrs
Silv3S Jan 10, 2023
efaacbf
revert matmul_utils.h
Silv3S Jan 10, 2023
60b2eec
codestyle
Silv3S Jan 10, 2023
3d266dd
adjust imports
Silv3S Jan 10, 2023
39edccd
add pbtxt file
Silv3S Jan 10, 2023
25659f4
Merge branch 'PaddlePaddle:develop' into remove_extra_matmul_attrs
Silv3S Jan 11, 2023
0dc52e0
100% matmul unit tests coverage
Silv3S Jan 11, 2023
a00c10e
trigger CI with minimal changes to develop
Silv3S Jan 11, 2023
274f399
adjust changes to develop
Silv3S Jan 11, 2023
bd57ba6
add fused_matmul op
Silv3S Jan 12, 2023
a825670
inherit base ops
Silv3S Jan 12, 2023
dbbeacc
add "v2"
Silv3S Jan 12, 2023
2276699
move OPMaker
Silv3S Jan 12, 2023
7f741b9
Gradually add fused_matmul files
Silv3S Jan 12, 2023
2fa89a8
second batch of fused_matmul changes
Silv3S Jan 12, 2023
bef0d26
split infershapes of matmul_v2 and fused_matmul
Silv3S Jan 12, 2023
d61c27b
2023
Silv3S Jan 17, 2023
862671a
Merge branch 'develop' into remove_extra_matmul_attrs
Silv3S Jan 17, 2023
f4b945a
inherit fused_matmul from matmul_v2
Silv3S Jan 17, 2023
b8dc210
Update paddle/phi/backends/onednn/onednn_reuse.h
Silv3S Jan 17, 2023
1ce346e
Update paddle/phi/kernels/fusion/onednn/fused_matmul_kernel.cc
Silv3S Jan 17, 2023
51e704a
resolve conflicts
Silv3S Jan 18, 2023
5cbb217
Merge branch 'develop' into remove_extra_matmul_attrs
Silv3S Jan 18, 2023
f0e2abe
codestyle
Silv3S Jan 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 76 additions & 11 deletions paddle/fluid/framework/ir/mkldnn/matmul_activation_mkldnn_fuse_pass.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
// Copyright (c) 2023 PaddlePaddle Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -26,7 +26,7 @@ using string::PrettyLogDetail;

void MatmulActivationMkldnnFusePass::ApplyImpl(Graph* graph) const {
auto act_types = GetSupportedActivations();
auto matmul_types = {"matmul", "matmul_v2"};
auto matmul_types = {"fused_matmul", "matmul", "matmul_v2"};

for (const auto& matmul_type : matmul_types)
for (auto& act_type : act_types) {
Expand Down Expand Up @@ -61,8 +61,17 @@ void MatmulActivationMkldnnFusePass::FuseMatmulAct(
GET_IR_NODE_FROM_SUBGRAPH(
activation_out, activation_out, matmul_act_pattern);

SetActivationAttrs(matmul->Op(), activation->Op(), act_type);
matmul->Op()->SetOutput("Out", {activation_out->Name()});
OpDesc* matmul_op = matmul->Op();

matmul_op->SetType("fused_matmul");
if (matmul_type == "matmul") {
matmul_op->SetAttr("trans_x", matmul_op->GetAttr("transpose_X"));
matmul_op->SetAttr("trans_y", matmul_op->GetAttr("transpose_Y"));
matmul_op->SetAttr("matmul_alpha", matmul_op->GetAttr("alpha"));
}

SetActivationAttrs(matmul_op, activation->Op(), act_type);
matmul_op->SetOutput("Out", {activation_out->Name()});

IR_NODE_LINK_TO(matmul, activation_out);
GraphSafeRemoveNodes(graph, {activation, matmul_out});
Expand All @@ -88,11 +97,6 @@ MatmulActivationMkldnnFusePass::MatmulActivationMkldnnFusePass() {
.AddInput("Y")
.IsTensor()
.End()
.AddInput(
"ResidualData") // Extra tensor used in matmul+elementwise_add fuse
.IsTensor()
.IsOptional()
.End()
.AddOutput("Out")
.IsTensor()
.End()
Expand All @@ -113,8 +117,24 @@ MatmulActivationMkldnnFusePass::MatmulActivationMkldnnFusePass() {
.AddInput("Y")
.IsTensor()
.End()
.AddInput(
"ResidualData") // Extra tensor used in matmul+elementwise_add fuse
.AddOutput("Out")
.IsTensor()
.End()
.AddAttr("trans_x")
.IsType<bool>()
.End()
.AddAttr("trans_y")
.IsType<bool>()
.End();

AddOpCompat(OpCompat("fused_matmul"))
.AddInput("X")
.IsTensor()
.End()
.AddInput("Y")
.IsTensor()
.End()
.AddInput("ResidualData")
.IsTensor()
.IsOptional()
.End()
Expand All @@ -126,6 +146,50 @@ MatmulActivationMkldnnFusePass::MatmulActivationMkldnnFusePass() {
.End()
.AddAttr("trans_y")
.IsType<bool>()
.End()
.AddAttr("matmul_alpha")
.IsType<float>()
.IsOptional()
.End()
.AddAttr("fuse_activation")
.IsType<std::string>()
.IsOptional()
.End()
.AddAttr("fuse_alpha")
.IsType<float>()
.IsOptional()
.End()
.AddAttr("fuse_beta")
.IsType<float>()
.IsOptional()
.End()
.AddAttr("fused_output_scale")
.IsType<float>()
.IsOptional()
.End()
.AddAttr("fused_reshape_X")
.IsType<std::vector<int>>()
.IsOptional()
.End()
.AddAttr("fused_transpose_X")
.IsType<std::vector<int>>()
.IsOptional()
.End()
.AddAttr("fused_reshape_Y")
.IsType<std::vector<int>>()
.IsOptional()
.End()
.AddAttr("fused_transpose_Y")
.IsType<std::vector<int>>()
.IsOptional()
.End()
.AddAttr("fused_reshape_Out")
.IsType<std::vector<int>>()
.IsOptional()
.End()
.AddAttr("fused_transpose_Out")
.IsType<std::vector<int>>()
.IsOptional()
.End();

AddOpCompat(OpCompat("abs"))
Expand Down Expand Up @@ -279,6 +343,7 @@ REGISTER_PASS(matmul_activation_mkldnn_fuse_pass,
REGISTER_PASS_CAPABILITY(matmul_activation_mkldnn_fuse_pass)
.AddCombination(
paddle::framework::compatible::OpVersionComparatorCombination()
.EQ("fused_matmul", 0)
.LE("matmul", 1)
.EQ("matmul_v2", 0)
.EQ("abs", 0)
Expand Down
76 changes: 74 additions & 2 deletions paddle/fluid/framework/ir/mkldnn/matmul_elementwise_add_mkldnn_fuse_pass.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
// Copyright (c) 2023 PaddlePaddle Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
Expand All @@ -25,7 +25,7 @@ namespace ir {
using string::PrettyLogDetail;

void MatmulElementwiseAddMKLDNNFusePass::ApplyImpl(Graph* graph) const {
auto matmul_types = {"matmul", "matmul_v2"};
auto matmul_types = {"fused_matmul", "matmul", "matmul_v2"};
auto matmul_as_x = {true, false};

for (const auto& matmul_type : matmul_types)
Expand Down Expand Up @@ -65,6 +65,12 @@ void MatmulElementwiseAddMKLDNNFusePass::FuseMatmulElementwiseAdd(
return;
}

matmul->Op()->SetType("fused_matmul");
if (matmul_type == "matmul") {
matmul->Op()->SetAttr("trans_x", matmul->Op()->GetAttr("transpose_X"));
matmul->Op()->SetAttr("trans_y", matmul->Op()->GetAttr("transpose_Y"));
matmul->Op()->SetAttr("matmul_alpha", matmul->Op()->GetAttr("alpha"));
}
matmul->Op()->SetInput("ResidualData", {elementwise_addend->Name()});
matmul->Op()->SetOutput("Out", {elementwise_add_out->Name()});

Expand Down Expand Up @@ -125,6 +131,71 @@ MatmulElementwiseAddMKLDNNFusePass::MatmulElementwiseAddMKLDNNFusePass() {
.IsType<bool>()
.End();

AddOpCompat(OpCompat("fused_matmul"))
.AddInput("X")
.IsTensor()
.End()
.AddInput("Y")
.IsTensor()
.End()
.AddInput("ResidualData")
.IsTensor()
.IsOptional()
.End()
.AddOutput("Out")
.IsTensor()
.End()
.AddAttr("trans_x")
.IsType<bool>()
.End()
.AddAttr("trans_y")
.IsType<bool>()
.End()
.AddAttr("matmul_alpha")
.IsType<float>()
.IsOptional()
.End()
.AddAttr("fuse_activation")
.IsType<std::string>()
.IsOptional()
.End()
.AddAttr("fuse_alpha")
.IsType<float>()
.IsOptional()
.End()
.AddAttr("fuse_beta")
.IsType<float>()
.IsOptional()
.End()
.AddAttr("fused_output_scale")
.IsType<float>()
.IsOptional()
.End()
.AddAttr("fused_reshape_X")
.IsType<std::vector<int>>()
.IsOptional()
.End()
.AddAttr("fused_transpose_X")
.IsType<std::vector<int>>()
.IsOptional()
.End()
.AddAttr("fused_reshape_Y")
.IsType<std::vector<int>>()
.IsOptional()
.End()
.AddAttr("fused_transpose_Y")
.IsType<std::vector<int>>()
.IsOptional()
.End()
.AddAttr("fused_reshape_Out")
.IsType<std::vector<int>>()
.IsOptional()
.End()
.AddAttr("fused_transpose_Out")
.IsType<std::vector<int>>()
.IsOptional()
.End();

AddOpCompat(OpCompat("elementwise_add"))
.AddInput("X")
.IsTensor()
Expand All @@ -149,6 +220,7 @@ REGISTER_PASS(matmul_elementwise_add_mkldnn_fuse_pass,
REGISTER_PASS_CAPABILITY(matmul_elementwise_add_mkldnn_fuse_pass)
.AddCombination(
paddle::framework::compatible::OpVersionComparatorCombination()
.EQ("fused_matmul", 0)
.LE("matmul", 1)
.EQ("matmul_v2", 0)
.LE("elementwise_add", 1));
76 changes: 74 additions & 2 deletions paddle/fluid/framework/ir/mkldnn/matmul_transpose_reshape_mkldnn_fuse_pass.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
// Copyright (c) 2023 PaddlePaddle Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
Expand All @@ -24,7 +24,7 @@ namespace ir {
using string::PrettyLogDetail;

void MatmulTransposeReshapeMKLDNNPass::ApplyImpl(Graph *graph) const {
auto matmul_types = {"matmul", "matmul_v2"};
auto matmul_types = {"fused_matmul", "matmul", "matmul_v2"};

for (const auto &matmul_type : matmul_types) {
Fuse(graph, matmul_type);
Expand Down Expand Up @@ -84,6 +84,12 @@ void MatmulTransposeReshapeMKLDNNPass::Fuse(
}

OpDesc *matmul_desc = matmul_op->Op();
matmul_desc->SetType("fused_matmul");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There have some broadcast computational logic difference between matmul_v1 and matmul_v2， the code as follows

import paddle 
data1 = paddle.rand((3, 1, 10, 10))
data2 = paddle.rand((1, 2, 10, 10))


# The paddle.matmul (matmul_v2)
paddle.matmul(data1, data2).shape # [3, 2, 10, 10]


# The fluid.layers.matmul (matmul_v1)
paddle.fluid.layers.matmul(data1, data2).shape # 直接报错
# ValueError: (InvalidArgument) The batch size of the two matrices should be equal, or at least one is zero.
# But received X's shape: [3, 10, 10], Y's shape: [2, 10, 10]

Is there some problems when replacing directly ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, I don't think that it could be problem. For some time oneDNN version of both matmul and matmul_v2 are already using same code #44640, #48162. Declared attributes of these two are mostly the same. Only difference is alpha not supported in matmul_v2, and transpose is renamed to trans).

Goal of this PR is just to extract fusion logic from base op and base kernel. Fused op is superset, which has all extra attributes declared in OPMaker and fused kernel has implemented support for handling fusion logic.

I've adjusted all fuse pass unit test to work with fused_matmul. Also this PR has been checked in our internal validation and it didn't report any accuracy drop.

if (matmul_type == "matmul") {
matmul_desc->SetAttr("trans_x", matmul_desc->GetAttr("transpose_X"));
matmul_desc->SetAttr("trans_y", matmul_desc->GetAttr("transpose_Y"));
matmul_desc->SetAttr("matmul_alpha", matmul_desc->GetAttr("alpha"));
}
matmul_desc->SetOutput("Out", {reshape_out->Name()});
matmul_desc->SetAttr("fused_reshape_Out", reshape_shape);
matmul_desc->SetAttr("fused_transpose_Out", transpose_axis);
Expand Down Expand Up @@ -149,6 +155,71 @@ MatmulTransposeReshapeMKLDNNPass::MatmulTransposeReshapeMKLDNNPass() {
.IsType<bool>()
.End();

AddOpCompat(OpCompat("fused_matmul"))
.AddInput("X")
.IsTensor()
.End()
.AddInput("Y")
.IsTensor()
.End()
.AddInput("ResidualData")
.IsTensor()
.IsOptional()
.End()
.AddOutput("Out")
.IsTensor()
.End()
.AddAttr("trans_x")
.IsType<bool>()
.End()
.AddAttr("trans_y")
.IsType<bool>()
.End()
.AddAttr("matmul_alpha")
.IsType<float>()
.IsOptional()
.End()
.AddAttr("fuse_activation")
.IsType<std::string>()
.IsOptional()
.End()
.AddAttr("fuse_alpha")
.IsType<float>()
.IsOptional()
.End()
.AddAttr("fuse_beta")
.IsType<float>()
.IsOptional()
.End()
.AddAttr("fused_output_scale")
.IsType<float>()
.IsOptional()
.End()
.AddAttr("fused_reshape_X")
.IsType<std::vector<int>>()
.IsOptional()
.End()
.AddAttr("fused_transpose_X")
.IsType<std::vector<int>>()
.IsOptional()
.End()
.AddAttr("fused_reshape_Y")
.IsType<std::vector<int>>()
.IsOptional()
.End()
.AddAttr("fused_transpose_Y")
.IsType<std::vector<int>>()
.IsOptional()
.End()
.AddAttr("fused_reshape_Out")
.IsType<std::vector<int>>()
.IsOptional()
.End()
.AddAttr("fused_transpose_Out")
.IsType<std::vector<int>>()
.IsOptional()
.End();

AddOpCompat(OpCompat("transpose2"))
.AddInput("X")
.IsTensor()
Expand Down Expand Up @@ -189,6 +260,7 @@ REGISTER_PASS(matmul_transpose_reshape_mkldnn_fuse_pass,
REGISTER_PASS_CAPABILITY(matmul_transpose_reshape_mkldnn_fuse_pass)
.AddCombination(
paddle::framework::compatible::OpVersionComparatorCombination()
.EQ("fused_matmul", 0)
.LE("matmul", 1)
.EQ("matmul_v2", 0)
.EQ("transpose2", 0)
Expand Down
17 changes: 16 additions & 1 deletion paddle/fluid/framework/ir/mkldnn/operator_scale_onednn_fuse_pass.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
// Copyright (c) 2023 PaddlePaddle Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -27,6 +27,7 @@ using string::PrettyLogDetail;
void FuseOperatorScaleOneDNNPass::ApplyImpl(Graph *graph) const {
const std::vector<std::string> fusable_ops{
"fc",
"fused_matmul",
"matmul",
"matmul_v2",
"elementwise_add",
Expand Down Expand Up @@ -85,6 +86,19 @@ void FuseOperatorScaleOneDNNPass::FuseScale(Graph *graph,
scale = *(scale_tensor->data<float>());
}

if (op_type == "matmul") {
operator_op->Op()->SetType("fused_matmul");
operator_op->Op()->SetAttr("trans_x",
operator_op->Op()->GetAttr("transpose_X"));
operator_op->Op()->SetAttr("trans_y",
operator_op->Op()->GetAttr("transpose_Y"));
operator_op->Op()->SetAttr("matmul_alpha",
operator_op->Op()->GetAttr("alpha"));
}
if (op_type == "matmul_v2") {
operator_op->Op()->SetType("fused_matmul");
}

operator_op->Op()->SetAttr("fused_output_scale", scale);
operator_op->Op()->SetOutput("Out", {scale_out->Name()});

Expand All @@ -111,6 +125,7 @@ REGISTER_PASS_CAPABILITY(operator_scale_onednn_fuse_pass)
.AddCombination(
paddle::framework::compatible::OpVersionComparatorCombination()
.EQ("fc", 0)
.EQ("fused_matmul", 0)
.LE("matmul", 1)
.EQ("matmul_v2", 0)
.LE("elementwise_add", 1)
Expand Down
Loading