Sls - Stochastic Line Search (NeurIPS2019) [paper][video]

Train faster and better with the SLS optimizer. The following 3 steps are there for getting started.

1. Installation

pip install git+https://github.com/IssamLaradji/sls.git

2. Usage

Use Sls in your code by adding the following script.

import sls
opt = sls.Sls(model.parameters())

for epoch in range(100):
      # create loss closure
      closure = lambda : torch.nn.MSELoss()(model(X), y)

      # update parameters
      opt.zero_grad()
      loss = opt.step(closure=closure)

3. Experiments

Install the experiment requirements pip install -r requirements.txt

2.1 MNIST

python trainval.py -e mnist -sb ../results -d ../data -r 1

where -e is the experiment group, -sb is the result directory, and -d is the dataset directory.

2.2 Cifar100 experiment

python trainval.py -e cifar100 -sb ../results -d ../data -r 1

3. Results

3.1 Launch Jupyter by running the following on terminal,

jupyter nbextension enable --py widgetsnbextension --sys-prefix
jupyter notebook

3.2 On a Jupyter cell, run the following script,

from haven import haven_jupyter as hj
from haven import haven_results as hr
from haven import haven_utils as hu

# path to where the experiments got saved
savedir_base = '../results'

# filter exps
filterby_list = [{'dataset':'cifar100', 'opt':{'c':0.5}}, 
                 {'dataset':'cifar100', 'opt':{'name':'adam'}}]
                 
# get experiments
rm = hr.ResultManager(savedir_base=savedir_base, 
                      filterby_list=filterby_list, 
                      verbose=0)
                      
# dashboard variables
legend_list = ['opt.name']
title_list = ['dataset', 'model']
y_metrics = ['train_loss', 'val_acc']

# launch dashboard
hj.get_dashboard(rm, vars(), wide_display=True)

Citation

@inproceedings{vaswani2019painless,
  title={Painless stochastic gradient: Interpolation, line-search, and convergence rates},
  author={Vaswani, Sharan and Mishkin, Aaron and Laradji, Issam and Schmidt, Mark and Gidel, Gauthier and Lacoste-Julien, Simon},
  booktitle={Advances in Neural Information Processing Systems},
  pages={3727--3740},
  year={2019}
}

It is a collaborative work between labs at MILA, Element AI, and UBC.