This repository was archived by the owner on Jan 24, 2024. It is now read-only.
Add adam optimizer #36
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
src/function.h
Outdated
| float beta2_; | ||
| // store the following items in order: first moment, second moment, beta1_pow, | ||
| // beta2_pow | ||
| std::unordered_map< |
Collaborator
There was a problem hiding this comment.
I feel the following four hyperparameters should be owned by Variable.
-
Hyperparameter should be strongly associated with parameters.
- When saving the parameters, we also need to save the parameters' hyperparameter.
- The lifetime of a hyperparameter should be the same as the parameter, not the optimizer. For example, when we try to optimize the network with
adam1first, then we want to switch to anotheradam2with a different set up(differentbeta1_,beta2_for example), the current implementation would require copying states fromadam1toadam2
-
In the future, I am planning to remove variable's name. The map from
stringtoVariableHandlehas to be changed by then.
Collaborator
There was a problem hiding this comment.
How about adding something like Variable::FirstMoment()?
Collaborator
Author
There was a problem hiding this comment.
Totally agree. I choose to use a map to store different optimizer's hyperparams so that we can conveniently add other optimizer that use hypers. Code has been modified.
tonyyang-svail
approved these changes
Jun 27, 2018
Collaborator
tonyyang-svail
left a comment
tonyyang-svail
left a comment
There was a problem hiding this comment.
Look good to me!
|
|
||
| #include <memory> | ||
| #include <string> | ||
| #include <vector> |
Collaborator
There was a problem hiding this comment.
mnior change: add #include <unordered_map>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.