You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@mratsim I'm curious about adding momentum to SGD (largely to avoid doing any actual work in my own Nim projects, ha). Would you want to do it in the same way as PyTorch/Tensorflow? That is, both libraries provide a single "SGD" optimizer with a momentum parameter, where for momentum > 0 momentum is applied, and where momentum == 0 it acts as simple SGD. They then also implement a "Nesterov" boolean, where true applies Nesterov momentum instead of regular momentum. Or do you envision a different implementation?
* Add momentum to SGD.
- I've retained an older non-momentum version of SGD for backcompatibility. Storing moments requires
a variable SGD object, and most code written for arraymancer prior to this more than likely defines
itsoptimizers with `let` since that is how it is done in the examples.
* Move the old moment update to before the weight update.
- This reording doesn't change the function of update(), but it does make it easier
to implement Nesterov momentum.
* Separate SGD with momentum into its own object.
- This preserves backwards compatibility with old `let optim` declared SGD optimizers.
* Add (optional) Nesterov momentum to the SGDMomentum optimizer.
* Add learning rate decay to SGDMomentum.
* Add tests for SGD with momentum.
* Add documentation to SGD.
* Remove an extraneous modification to newSGD.
Currently only stochastic gradient descent is supported, at the very minimum it would be nice to support:
The text was updated successfully, but these errors were encountered: