Steps scheduling features for LR Schedulers?

[abetteridea]

As far as I can see after an extensive research (discussions, not code), the options for steps scheduling during training are currently very limited.

Cosine with restarts can be tuned with lr_scheduler_num_cycles for the restarts to happen more (or less) often than every epoch (default), which is great.

But what about Constant steps scheduling? How do I set a LR of 5e-5 for the first 1000 steps, LR of 5e-6 for the next 1000 and LR of 5e-7 till the end? A syntax like 5e-5:1000, 5e-6:2000, 5e-7 would've been so helpful! I actually can't believe there are no settings for Constant or Linear steps schedules. Am I missing something?

P.S. Cosine LR scheduler curve settings would've been incredible, but I'm asking too much already I think!

[abetteridea]

There is lr_scheduler_power for Polynomial LR Scheduler though, which is amazing

[slashedstar]

#271
You can even create custom schedulers by editing the code, which would be needed to do what you want, though MultiStepLR is close to that, you just can't change the gamma
https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.MultiStepLR.html#torch.optim.lr_scheduler.MultiStepLR