Add simple Schedulers

[DhairyaLGandhi]

Simple design for scheduling learning rate decay using the Optimiser interface

[gxyd]

Is the intention here to add other Schedulers as well that Pytorch implements?

* LambdaLR
* MultiplicativeLR (this PR?)
* StepLR
* MultiStepLR
* ExponentialLR
* CosineAnnealingLR
* ReduceLROnPlateu
* CyclicLR
* OneCycleLR
* CosineAnnealingWarmRestarts

[DhairyaLGandhi]

Happy to have help with this! Having said that, the design of the optimisers seems to lend itself fine to adding schedulers. Note that we would want to have a fairly shallow type hierarchy.

We probably want to talk about what all features schedulers might need that can fall off from this design and what the limitations are.

[DhairyaLGandhi]

I find the method to find the current epoch number a bit janky, which I would love to improve upon as a start

[ToucheSir]

I think this should be reconciled with https://github.com/darsnack/ParameterSchedulers.jl somehow. cc @darsnack and @lorenzoh

[darsnack]

ParameterSchedulers.jl should have most of PyTorch's functionality already implemented. The only ones that I'd want to double check are ReduceLROnPlateu and CosineAnnealingWarmRestarts. Probably these are just syntactic sugar to compose the existing schedulers.

@CarloLucibello mentioned integrating ParameterSchedulers.jl into Flux. I would have already made a PR, but I ran head first into Flux's limited optimizer interface. The main issue is that apply! is called on a per parameter basis. So, it ends up getting called many times per iteration, meaning that any scheduler would need to know a priori the number of times it will be called. This is computable but hacky imo. You can't rely on overloading update! since that only works for implicit parameters and breaks composability (i.e. Flux.Optimise.Optimiser doesn't invoke update! on each sub-optimizer, it invokes apply!).

I am currently writing up what a future interface should look like, and I will post it soon. I'll follow up with a PR to Flux that implements it based on the discussion that follows that post.

PS: This really only presents a problem for Flux.train! where the training loop is inaccessible to the user.

[darsnack]

Looking at the source of this PR, it will hit the same issue that ScheduledOptim from ParameterSchedulers.jl did. o.current will end up getting updated more than once per iteration since apply! will be called once for every parameter. You could reconcile this by passing params(m) into the constructor and doing the scheduler logic only once per iteration, but that's too hacky.

[DhairyaLGandhi]

will end up getting updated more than once per iteration

I'm aware of the bug, which isn't difficult to fix but this was more for design discussion than final implementation. You'll notice the janky name :)

[darsnack]

Good to hear — a design discussion is exactly what I think is needed too.

[DhairyaLGandhi]

Do post your thoughts here @darsnack, that's kind of the motivation of the PR.

The question on composition is that we can make it safe to call apply on a number of parameters together, but it's not very intuitive. To be clear, i want to move these over to optimisers.jl now, so we have a unified interface. It's written with that in mind.

[darsnack]

To be clear, i want to move these over to optimisers.jl now, so we have a unified interface. It's written with that in mind.

Perfect, I was thinking the exact same thing.

[ToucheSir]

I do think the interaction between learning rate schedule and optimizer will be very different moving from the current Flux interface to the mostly stateless one in Optimisers.jl. For example, ScheduledOptim.update_func can no longer be a purely mutating function, and I wonder if a PyTorch-esque interface is the best way to go. I know I keep mentioning it, but it's illustrative to see what a mutation-free library like Optax does here.

[darsnack]

I posted my thoughts in a discussion. I'm interested to hear how people think the interface can be improved. Personally, I felt that I'm still unsatisfied with hyperparameter accesses, but I can't figure out anything better.