Skip to content

Added Scaled Exponential Linear Unit Activation Function #9027

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Sep 6, 2023
Merged
Prev Previous commit
Next Next commit
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
  • Loading branch information
pre-commit-ci[bot] committed Sep 2, 2023
commit cf66ff8f368c62d3651027425fe24201e6e009dd
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
Implements the Scaled Exponential Linear Unit or SELU function.

The function takes a vector of K real numbers and two real numbers alpha(default = 1.6732) and lambda (default = 1.0507)
as input and then applies the SELU function to each element of the vector. SELU is a self-normalizing activation function.
It is a variant of the ELU. The main advantage of SELU is that we can be sure that the output will always be standardized
as input and then applies the SELU function to each element of the vector. SELU is a self-normalizing activation function.
It is a variant of the ELU. The main advantage of SELU is that we can be sure that the output will always be standardized
due to its self-normalizing behavior. That means there is no need to include Batch-Normalization layers.

References :
Expand All @@ -12,7 +12,10 @@

import numpy as np

def scaled_exponential_linear_unit(vector: np.ndarray, alpha : float = 1.6732, _lambda : float = 1.0507) -> np.ndarray:

def scaled_exponential_linear_unit(
vector: np.ndarray, alpha: float = 1.6732, _lambda: float = 1.0507
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why _lambda has an underscore? Is the user not meant to change this coefficient?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, firstly lambda is a reserved keyword in python. Secondly both alpha and lambda are fixed constants.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The user can change these values, that may yield slightly different behaviour from the function, but the default values given to them are alpha: float = 1.6732, _lambda: float = 1.0507.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, firstly lambda is a reserved keyword in python.

Oh yeah, duh 🤦

Could you rename the variable to something like lambda_ instead? Having an underscore at the start of a variable name generally signifies that the user isn't supposed to use it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah. Yes for sure, I'll get it done right away!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

) -> np.ndarray:
"""
Applies the Scaled Exponential Linear Unit function to each element of the vector.
Parameters : vector : np.ndarray
Expand All @@ -30,6 +33,8 @@ def scaled_exponential_linear_unit(vector: np.ndarray, alpha : float = 1.6732, _
"""
return _lambda * np.where(vector > 0, vector, alpha * (np.exp(vector) - 1))


if __name__ == "__main__":
import doctest
doctest.testmod()

doctest.testmod()