Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite activation function #5465

Merged
merged 110 commits into from
Jul 29, 2021
Merged
Changes from 1 commit
Commits
Show all changes
110 commits
Select commit Hold shift + click to select a range
c7b9a1b
add activation
MARD1NO Jul 12, 2021
c68202b
rename swish to silu
MARD1NO Jul 12, 2021
fddeb30
add selu
MARD1NO Jul 12, 2021
7788ddb
add four activation op
MARD1NO Jul 12, 2021
3178171
add softsign test
MARD1NO Jul 13, 2021
0718510
add silu mish selu softsign
MARD1NO Jul 13, 2021
5f72453
Add softsign docs
MARD1NO Jul 13, 2021
1774113
Add functional impl
MARD1NO Jul 13, 2021
ea36901
small fix for softsign backward
MARD1NO Jul 13, 2021
4dd7261
remove flow.mish test
MARD1NO Jul 13, 2021
cf90d96
add silu module test
MARD1NO Jul 13, 2021
0ae60de
add selu test
MARD1NO Jul 13, 2021
d72f836
fix docs
MARD1NO Jul 13, 2021
9529123
fix softsign docs
MARD1NO Jul 13, 2021
ae388b5
fix format
MARD1NO Jul 13, 2021
2614225
fix static cast
MARD1NO Jul 13, 2021
10ebada
Merge branch 'master' into rewrite_activation_function
MARD1NO Jul 14, 2021
750ed8b
merge master functional api yaml
MARD1NO Jul 14, 2021
0db666e
add torch style unittest
MARD1NO Jul 14, 2021
53e1b75
Remove assert and add torch unittest
MARD1NO Jul 14, 2021
c81fde4
add tensor def
MARD1NO Jul 14, 2021
d4c02a5
Merge branch 'master' into rewrite_activation_function
MARD1NO Jul 14, 2021
5f6b7d6
Merge branch 'master' into rewrite_activation_function
MARD1NO Jul 14, 2021
0a648ad
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 14, 2021
cc55d7b
remove softsign test temporary
MARD1NO Jul 14, 2021
0912f5b
Merge branch 'rewrite_activation_function' of https://github.com/Onef…
MARD1NO Jul 14, 2021
8dfcfbb
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 14, 2021
eee2d39
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 14, 2021
6260022
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 14, 2021
ff1f320
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 15, 2021
a80c051
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 15, 2021
968f509
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 15, 2021
2073874
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 15, 2021
c94f726
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 15, 2021
db3c8aa
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 15, 2021
381af06
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 15, 2021
4783fbc
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 15, 2021
48bb33a
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 15, 2021
7a1f65a
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 15, 2021
3f63153
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 15, 2021
0f07be9
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 16, 2021
f0325db
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 16, 2021
5859938
Merge branch 'master' into rewrite_activation_function
MARD1NO Jul 19, 2021
b913a5b
add return maybe ok
MARD1NO Jul 19, 2021
b917cba
Merge branch 'rewrite_activation_function' of https://github.com/Onef…
MARD1NO Jul 19, 2021
2f64bbe
Merge branch 'master' into rewrite_activation_function
MARD1NO Jul 19, 2021
3661fe6
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 19, 2021
e69ae36
Merge branch 'master' into rewrite_activation_function
MARD1NO Jul 20, 2021
6105284
migrate nn ops to single_client dir
MARD1NO Jul 20, 2021
e13f47f
migrate unittest
MARD1NO Jul 20, 2021
7c71113
remove lazy unittest
MARD1NO Jul 20, 2021
2ea2a63
add unittest
MARD1NO Jul 20, 2021
d5ba766
Merge branch 'master' into rewrite_activation_function
MARD1NO Jul 20, 2021
b016845
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 20, 2021
e66923b
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 20, 2021
435de4d
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 20, 2021
fcc2ddd
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 20, 2021
e898fc0
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 20, 2021
b2085c9
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 20, 2021
a518e1e
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 20, 2021
2112869
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 20, 2021
6b84a62
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 20, 2021
22db7ac
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 20, 2021
f2b9101
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 21, 2021
7f9d745
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 21, 2021
d480187
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 21, 2021
f97208f
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 21, 2021
3a9cd5f
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 22, 2021
b40f83b
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 22, 2021
f8fa56a
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 22, 2021
3410182
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 22, 2021
1e965cc
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 22, 2021
8686e48
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 22, 2021
f3c9990
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 22, 2021
98e982a
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 22, 2021
9e12026
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 23, 2021
8ce4582
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 23, 2021
f6258cb
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 23, 2021
2f5c0d1
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 23, 2021
f81b821
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 23, 2021
8b6b91d
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 23, 2021
7474d80
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 23, 2021
4f35021
Merge branch 'master' into rewrite_activation_function
MARD1NO Jul 27, 2021
766677d
fix to new directory
MARD1NO Jul 28, 2021
89e2e5a
Merge branch 'rewrite_activation_function' of https://github.com/Onef…
MARD1NO Jul 28, 2021
dea0f21
Remove useless docs and single client test
MARD1NO Jul 28, 2021
f22a3a6
add doc
MARD1NO Jul 28, 2021
e08d768
fix docs
MARD1NO Jul 28, 2021
1561899
add docs in oneflow and tensor namespace
MARD1NO Jul 28, 2021
062a164
add torch autotest
MARD1NO Jul 28, 2021
0c78888
fix to new autotest
MARD1NO Jul 28, 2021
e672d90
remove outdated python code
MARD1NO Jul 28, 2021
d12916e
Merge branch 'master' into rewrite_activation_function
MARD1NO Jul 28, 2021
07939a1
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 28, 2021
44fd005
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 28, 2021
f435ed2
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 28, 2021
67d7c06
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 28, 2021
ba2b106
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 28, 2021
6015388
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 28, 2021
33ca9ae
remove useless docs
MARD1NO Jul 29, 2021
4ba086b
Merge branch 'rewrite_activation_function' of https://github.com/Onef…
MARD1NO Jul 29, 2021
f52c625
enlarge unittest tolerance
MARD1NO Jul 29, 2021
3fce373
Add static cast for const value
MARD1NO Jul 29, 2021
c1832e4
skip softsign unittest
MARD1NO Jul 29, 2021
990db45
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 29, 2021
e244958
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 29, 2021
22557ac
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 29, 2021
72584ba
skip tensor softsign
MARD1NO Jul 29, 2021
3cb4008
Merge branch 'rewrite_activation_function' of https://github.com/Onef…
MARD1NO Jul 29, 2021
7541a0c
Merge branch 'master' into rewrite_activation_function
oneflow-ci-bot Jul 29, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix static cast
MARD1NO committed Jul 13, 2021
commit 261422548583ae733bd9750af10c50ae3995a7c7
4 changes: 2 additions & 2 deletions oneflow/user/kernels/activation_kernels.h
Original file line number Diff line number Diff line change
@@ -128,9 +128,9 @@ struct MishGradFunctor {
OF_DEVICE_FUNC explicit MishGradFunctor() {}
OF_DEVICE_FUNC T operator()(T x, T dy) const {
T sp = log(static_cast<T>(1) + exp(x));
T grad_sp = 1 - exp(-sp);
T grad_sp = static_cast<T>(1) - exp(-sp);
T tsp = (exp(sp) - exp(-sp)) / (exp(sp) + exp(-sp));
T grad_tsp = (1 - tsp * tsp) * grad_sp;
T grad_tsp = (static_cast<T>(1) - tsp * tsp) * grad_sp;
return dy * (x * grad_tsp + tsp);
}
};