Skip to content

Conversation

MingkangW
Copy link
Contributor

The type of add ops output is activation, it should annotate by QAnnotateKind.ACTIVATION. If not, the graph will cast int32 into int8 directly without quantized, when quantize resnet. It will product overflow error, and it wont prompt when infer the model. The resnet18_v1 test case is below.
before fixed
%703 = add(%701, %702) /* ty=Tensor[(1, 512, 7, 7), int32] /;
%704 = nn.relu(%703) /
ty=Tensor[(1, 512, 7, 7), int32] /;
%705 = cast(%704, dtype="int8") /
ty=Tensor[(1, 512, 7, 7), int8] /;
%706 = annotation.stop_fusion(%705) /
ty=Tensor[(1, 512, 7, 7), int8] /;
after fixed
%443 = add(%441, %442) /
ty=Tensor[(1, 512, 7, 7), int32] /;
%444 = nn.relu(%443) /
ty=Tensor[(1, 512, 7, 7), int32] /;
%445 = cast(%444, dtype="int64") /
ty=Tensor[(1, 512, 7, 7), int64] /;
%446 = fixed_point_multiply(%445, multiplier=1439683968, shift=-2) /
ty=Tensor[(1, 512, 7, 7), int64] /;
%447 = clip(%446, a_min=-127f, a_max=127f) /
ty=Tensor[(1, 512, 7, 7), int64] /;
%448 = cast(%447, dtype="int32") /
ty=Tensor[(1, 512, 7, 7), int32] /;
%449 = cast(%448, dtype="int8") /
ty=Tensor[(1, 512, 7, 7), int8] /;
%450 = annotation.stop_fusion(%449) /
ty=Tensor[(1, 512, 7, 7), int8] */;

The type of add op`s output is activation, it should annotate by QAnnotateKind.ACTIVATION. If not, the graph will cast int32 into int8 directly without quantized, when quantize resnet.
@tvm-bot
Copy link
Collaborator

tvm-bot commented Aug 11, 2023

Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.

  • No users to tag found in teams: quantize See #10317 for details

Generated by tvm-bot

cbalint13

This comment was marked as outdated.

Copy link
Contributor

@cbalint13 cbalint13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MingkangW ,
Can also add a small test case ?

@MingkangW
Copy link
Contributor Author

@MingkangW ,
Can also add a small test case ?

Sure, I will do it

@MingkangW
Copy link
Contributor Author

@cbalint13
I have added test cases. If the annotation for add is not modified to ACTIVATION, the test case will not pass.

Copy link
Contributor

@cbalint13 cbalint13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MingkangW ,
LGTM, thank you !

@MingkangW MingkangW requested a review from cbalint13 August 13, 2023 03:20
@MingkangW
Copy link
Contributor Author

MingkangW commented Aug 14, 2023

PTAL @cbalint13 @tqchen

@MingkangW
Copy link
Contributor Author

cc: @masahi

@masahi masahi merged commit 63592fd into apache:main Aug 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants