fusion hard_swish in int8 caused accuracy drop

[lidanqing-intel]

1）PaddlePaddle版本：2.1

[paddle-bot-old]

您好，我们已经收到了您的问题，会安排技术人员尽快解答您的问题，请耐心等待。请您再次检查是否提供了清晰的问题描述、复现代码、环境&版本、报错信息等。同时，您也可以通过查看官网API文档、常见问题、历史Issue、AI社区来寻求解答。祝您生活愉快～

Hi! We've received your issue and please be patient to get responded. We will arrange technicians to answer your questions as soon as possible. Please make sure that you have posted enough message to demo your request. You may also check out the API，FAQ，Github Issue and AI community to get the answer.Have a nice day!

[sfraczek]

Hi,
I found some divergence between OneDNN and PaddlePaddle hard swish implementations. The paper describes hard_swish as having threshold equal 3. Documentations for both projects PaddlePaddle and OneDNN have the same value 3. However, in PaddlePaddle code, the default value for threshold is 6.

OneDNN

Paddle

https://www.paddlepaddle.org.cn/documentation/docs/en/api/paddle/nn/functional/hardswish_en.html#hardswish

Paddle/paddle/fluid/operators/activation_op.cc

Lines 809 to 837 in a37be82

	class HardSwishOpMaker : public framework::OpProtoAndCheckerMaker {
	public:
	void Make() override {
	AddInput("X", "Input of HardSwish operator");
	AddOutput("Out", "Output of HardSwish operator");
	AddAttr<float>("threshold", "The threshold parameter of HardSwish operator")
	.SetDefault(6.0f);
	AddAttr<float>("scale", "The scale parameter of HardSwish operator")
	.SetDefault(6.0f);
	AddAttr<float>("offset", "The offset parameter of HardSwish operator")
	.SetDefault(3.0f);
	AddAttr<bool>("use_mkldnn",
	"(bool, default false) Only used in mkldnn kernel")
	.SetDefault(false)
	.AsExtra();
	AddComment(R"DOC(
	HardSwish Activation Operator.
	The hard version of swish(https://arxiv.org/pdf/1905.02244.pdf).
	$$out = \frac{x * (min(max(0, x+offset), threshold))}{scale}$$
	The threshold and scale should be positive. The offset can be either positive or negative.
	The default parameters are set according to the above reference.
	It is recommended to use the defaults for this activation.
	)DOC");
	}
	};

@huangjun12 please take a look. Does it have to be 6 not 3? I wonder if this could be causing change in accuracy?

[lidanqing-intel]

@lidanqing-intel @baoachun

[sfraczek]

I found that actually paddle's hard_swish is same as oneDNN's just using different equation and thus different meaning for threshold.