Quasi Hyperbolic Rectified DEMON (Decaying Momentum) Adam/Amsgrad with AdaMod, Lookahead, iterate averaging, and decorrelated weight decay.
Also, other variants with Nostalgia (NosAdam), P (from PAdam), LaProp, and Hypergradient Descent (see HyperRanger and HyperRangerMod and others in optimizers.py)
-
Hyperxxx series optimizers implements hypergradient descent for dynamic learning rate updates. Some optimizers like HDQHSGDW implements hypergradient descent for all hyperparameters - beta, nu, lr. Unlike the original implementation (https://arxiv.org/abs/1703.04782, https://github.com/gbaydin/hypergradient-descent) they take care of the gradients due to the weight decay and other things. (I also implement state level lr so that lr for each parameters will be hypertuned through hypergradient descent separately instead of in the group level like in the original implementation)
-
LRangerMod uses Linear Warmup within Adam/AMSGrad based on the rule of thumb as in (https://arxiv.org/abs/1910.04209v1). Note Rectified Adam boils down to a fixed (not dynamic) form of learning rate scheduling similar to a linear warmup.
-
The file explains the parameters for each different synergistic optimizers.
from optimizers import DemonRanger
from dataloader import batcher # some random function to batch data
class config:
def __init__(self):
self.batch_size = ...
self.wd = ...
self.lr = ...
self.epochs = ...
config = config()
train_data = ...
step_per_epoch = count_step_per_epoch(train_data,config.batch_size)
model = module(stuff)
optimizer = DemonRanger(params=model.parameters(),
lr=config.lr,
weight_decay=config.wd,
epochs=config.epochs,
step_per_epoch=step_per_epoch,
IA_cycle=step_per_epoch)
IA_activate = False
for epoch in range(config.epochs):
batches = batcher(train_data, config.batch_size)
for batch in batches:
loss = do stuff
loss.backward()
optimizer.step(IA_activate=IA_activate)
# automatically enable IA (Iterate Averaging) near the end of training (when metric of your choice not improving for a while)
if (IA_patience running low) and IA_activate is False:
IA_activate = True
optimizer = DemonRanger(params=model.parameters(),
lr=config.lr,
betas=(0.9,0.999,0.999), # restore default AdamW betas
nus=(1.0,1.0), # disables QHMomentum
k=0, # disables lookahead
alpha=1.0,
weight_decay=config.wd,
IA=False, # disables Iterate Averaging
rectify=False, # disables RAdam Recitification
AdaMod=False, #disables AdaMod
use_demon=False, #disables Decaying Momentum (DEMON)
use_gc=False, #disables gradient centralization
amsgrad=False # disables amsgrad
)
# just do optimizer.step() when necessary
optimizer = DemonRanger(params=model.parameters(),
lr=config.lr,
betas=(0.9,0.999,0.999), # restore default AdamW betas
nus=(1.0,1.0), # disables QHMomentum
k=0, # disables lookahead
alpha=1.0,
weight_decay=config.wd,
IA=False, # disables Iterate Averaging
rectify=False, # disables RAdam Recitification
AdaMod=False #disables AdaMod
use_demon=False, #disables Decaying Momentum (DEMON)
use_gc=False, #disables gradient centralization
amsgrad=True # disables amsgrad
)
# just do optimizer.step() when necessary
optimizer = DemonRanger(params=model.parameters(),
lr=config.lr,
k=0, # disables lookahead
alpha=1.0,
weight_decay=config.wd,
IA=False, # disables Iterate Averaging
rectify=False, # disables RAdam Recitification
AdaMod=False, #disables AdaMod
use_demon=False, #disables Decaying Momentum (DEMON)
use_gc=False, #disables gradient centralization
amsgrad=False # disables amsgrad
)
# just do optimizer.step() when necessary
optimizer = DemonRanger(params=model.parameters(),
lr=config.lr,
betas=(0.9,0.999,0.999), # restore default AdamW betas
nus=(1.0,1.0), # disables QHMomentum
k=0, # disables lookahead
alpha=1.0,
weight_decay=config.wd,
IA=False, # disables Iterate Averaging
AdaMod=False, #disables AdaMod
use_demon=False, #disables Decaying Momentum (DEMON)
use_gc=False, #disables gradient centralization
amsgrad=False # disables amsgrad
)
# just do optimizer.step() when necessary
optimizer = DemonRanger(params=model.parameters(),
lr=config.lr,
betas=(0.9,0.999,0.999), # restore default AdamW betas
nus=(1.0,1.0), # disables QHMomentum
weight_decay=config.wd,
IA=False, # disables Iterate Averaging
AdaMod=False, #disables AdaMod
use_demon=False, #disables Decaying Momentum (DEMON)
use_gc=False, #disables gradient centralization
amsgrad=False # disables amsgrad
)
# just do optimizer.step() when necessary
optimizer = DemonRanger(params=model.parameters(),
lr=config.lr,
weight_decay=config.wd,
IA=False, # disables Iterate Averaging
AdaMod=False, #disables AdaMod
use_demon=False, #disables Decaying Momentum (DEMON)
use_gc=False, #disables gradient centralization
amsgrad=False # disables amsgrad
)
# just do optimizer.step() when necessary
optimizer = DemonRanger(params=model.parameters(),
lr=config.lr,
weight_decay=config.wd,
betas=(0.9,0.999,0.999), # restore default AdamW betas
nus=(1.0,1.0), # disables QHMomentum
k=0, # disables lookahead
alpha=1.0,
IA=False, # disables Iterate Averaging
rectify=False, # disables RAdam Recitification
AdaMod_bias_correct=False, #disables AdaMod bias corretion (not used originally)
use_demon=False #disables Decaying Momentum (DEMON)
use_gc=False #disables gradient centralization
amsgrad=False # disables amsgrad
)
# just do optimizer.step() when necessary
optimizer = DemonRanger(params=model.parameters(),
lr=config.lr,
weight_decay=config.wd,
betas=(0.9,0.999,0.999), # restore default AdamW betas
nus=(1.0,1.0), # disables QHMomentum
k=0, # disables lookahead
alpha=1.0,
IA=True, # enables Iterate Averaging
rectify=False, # disables RAdam Recitification
AdaMod=False, #disables AdaMod
use_demon=False, #disables Decaying Momentum (DEMON)
use_gc=False, #disables gradient centralization
amsgrad=False # disables amsgrad
)
# just do optimizer.step(IA_activate=IA_activate) when necessary (change IA_activate to True near the end of training based on some scheduling scheme or tuned hyperparameter--- alternative to learning rate scheduling)
optimizer = DemonRanger(params=model.parameters(),
lr=config.lr,
weight_decay=config.wd,
betas=(0.9,0.999,0.999), # restore default AdamW betas
nus=(1.0,1.0), # disables QHMomentum
k=5, # enables lookahead
alpha=0.88,
IA=True, # enables Iterate Averaging
rectify=False, # disables RAdam Recitification
AdaMod=False, #disables AdaMod
use_demon=False, #disables Decaying Momentum (DEMON)
use_gc=False, #disables gradient centralization
amsgrad=False # disables amsgrad
)
# just do optimizer.step(IA_activate=IA_activate) when necessary (change IA_activate to True near the end of training based on some scheduling scheme or tuned hyperparameter--- alternative to learning rate scheduling)
optimizer = DemonRanger(params=model.parameters(),
lr=config.lr,
weight_decay=config.wd,
epochs = config.epochs,
step_per_epoch = step_per_epoch,
betas=(0.9,0.999,0.999), # restore default AdamW betas
nus=(1.0,1.0), # disables QHMomentum
k=0, # disables lookahead
alpha=1.0,
IA=False, # enables Iterate Averaging
rectify=False, # disables RAdam Recitification
AdaMod=False, #disables AdaMod
AdaMod_bias_correct=False, #disables AdaMod bias corretion (not used originally)
use_demon=True, #enables Decaying Momentum (DEMON)
use_gc=False, #disables gradient centralization
amsgrad=False # disables amsgrad
)
# just do optimizer.step() when necessary
Use Variance Rectified DEMON QHAMSGradW with AdaMod, LookAhead, Iterate Averaging, and Gradient Centralization
optimizer = DemonRanger(params=model.parameters(),
lr=config.lr,
weight_decay=config.wd,
epochs=config.epochs,
step_per_epoch=step_per_epoch,
IA_cycle=step_per_epoch)
# just do optimizer.step(IA_activate=IA_activate) when necessary (change IA_activate to True near the end of training based on some scheduling scheme or tuned hyperparameter--- alternative to learning rate scheduling)
- Dense-sparse-Dense Training: https://arxiv.org/pdf/1607.04381.pdf
- Bayesian Deep Learning: SWAG/SWA
- Adam: https://arxiv.org/abs/1412.6980
- AMSGrad: https://arxiv.org/abs/1904.09237
- QHAdam: https://arxiv.org/abs/1810.06801
- Gradient Noise: https://arxiv.org/abs/1511.06807
- AdamW: https://arxiv.org/abs/1711.05101
- RAdam: https://arxiv.org/abs/1908.03265, https://github.com/LiyuanLucasLiu/RAdam
- More on RAdam: https://arxiv.org/abs/1910.04209v1
- Lookahead: https://arxiv.org/abs/1907.08610
- Ranger: https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer
- Gradient Centralization: https://arxiv.org/abs/2004.01461v2
- DEMON (Decaying Momentum): https://arxiv.org/abs/1910.04952
- AdaMod: https://arxiv.org/abs/1910.12249
- GAdam (Iterate Averaging): https://arxiv.org/abs/2003.01247, https://github.com/diegogranziol/Gadam
- Hypergradient Descent: https://arxiv.org/abs/1703.04782, https://github.com/gbaydin/hypergradient-descent
- Nostalgic Adam: https://arxiv.org/abs/1805.07557, https://github.com/andrehuang/NostalgicAdam-NosAdam
- PAdam: https://arxiv.org/abs/1806.06763, https://github.com/uclaml/Padam, https://arxiv.org/pdf/1901.09517.pdf
- LaProp: https://arxiv.org/abs/2002.04839