PR #18105: [Add Feature] - Warn user if softmax usage is wrong #18137
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![copybara-service[bot]](https://avatars.githubusercontent.com/in/44061?s=60&v=4){kind=small}
Imported from GitHub PR #18105 This is a utility function to check if the usage of softmax makes sense (new users make this mistake a lot). Applying softmax on a single neuron will make the model output ones everytime, there are too many Stackoverflow posts about this. In order to see this in action, please check [the gist](https://colab.research.google.com/gist/Frightera/fdcec020fff6ee9521ae2fd3eaed774d/checksoftmaxlastlayer.ipynb). This applies for any other layers (Conv2D etc.) where the applied axis (axis=-1 default) of softmax has only one unit. Copybara import of the project: -- 90c95b1 by Kaan Bıçakcı <46622558+Frightera@users.noreply.github.com>: Add last layer activation check for softmax -- 1cedb20 by Kaan Bıçakcı <46622558+Frightera@users.noreply.github.com>: Split logic for sequential and functional models -- 529f968 by Kaan Bıçakcı <46622558+Frightera@users.noreply.github.com>: Add tests for _check_last_layer_activation -- d1acddb by Kaan Bıçakcı <46622558+Frightera@users.noreply.github.com>: Update sequential check -- 8363016 by Kaan Bıçakcı <46622558+Frightera@users.noreply.github.com>: Update tests, logic and reformatting -- ebf16c3 by Kaan Bıçakcı <46622558+Frightera@users.noreply.github.com>: Update tests and the logic -- afc156a by Kaan Bıçakcı <46622558+Frightera@users.noreply.github.com>: Make validate_softmax_activation experimental Merging this change closes #18105 FUTURE_COPYBARA_INTEGRATE_REVIEW=#18105 from Frightera:last_layer_softmax_warn afc156a PiperOrigin-RevId: 533267082
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR #18105: [Add Feature] - Warn user if softmax usage is wrong
Imported from GitHub PR #18105
This is a utility function to check if the usage of softmax makes sense (new users make this mistake a lot). Applying softmax on a single neuron will make the model output ones everytime, there are too many Stackoverflow posts about this.
In order to see this in action, please check the gist.
This applies for any other layers (Conv2D etc.) where the applied axis (axis=-1 default) of softmax has only one unit.
Copybara import of the project:
--
90c95b1 by Kaan Bıçakcı 46622558+Frightera@users.noreply.github.com:
Add last layer activation check for softmax
--
1cedb20 by Kaan Bıçakcı 46622558+Frightera@users.noreply.github.com:
Split logic for sequential and functional models
--
529f968 by Kaan Bıçakcı 46622558+Frightera@users.noreply.github.com:
Add tests for _check_last_layer_activation
--
d1acddb by Kaan Bıçakcı 46622558+Frightera@users.noreply.github.com:
Update sequential check
--
8363016 by Kaan Bıçakcı 46622558+Frightera@users.noreply.github.com:
Update tests, logic and reformatting
--
ebf16c3 by Kaan Bıçakcı 46622558+Frightera@users.noreply.github.com:
Update tests and the logic
--
afc156a by Kaan Bıçakcı 46622558+Frightera@users.noreply.github.com:
Make validate_softmax_activation experimental
Merging this change closes #18105
FUTURE_COPYBARA_INTEGRATE_REVIEW=#18105 from Frightera:last_layer_softmax_warn afc156a