Skip to content

Commit af662ac

Browse files
fsschneiderfsschneider
committed
update to version 1.1.0
1 parent 304d4fe commit af662ac

File tree

1,955 files changed

+12739
-3156063
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

1,955 files changed

+12739
-3156063
lines changed

‎README.md‎

Lines changed: 33 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -1,98 +1,50 @@
1-
[![Documentation Status](https://readthedocs.org/projects/deepobs/badge/?version=latest)](http://deepobs.readthedocs.io/?badge=latest)
2-
3-
41
# DeepOBS - A Deep Learning Optimizer Benchmark Suite
52

6-
DeepOBS is a benchmarking suite that drastically simplifies, automates and improves the evaluation of deep learning optimizers.
7-
8-
It can evaluate the performance of new optimizers on a variety of **real-world test problems** and automatically compare them with **realistic baselines**.
9-
10-
The full documentation is available on readthedocs: https://deepobs.readthedocs.io/
11-
12-
The paper describing DeepOBS has been accepted for ICLR 2019 and can be found here:
13-
https://openreview.net/forum?id=rJg6ssC5Y7
14-
15-
Currently we provide DeepOBS only for TensorFlow, but plan to provide a PyTorch version soon. In the meantime, PyTorch users can still use parts of DeepOBS such as the data preprocessing scripts or the visualization features.
16-
17-
## Quick Start Guide
18-
19-
### Install Deep OBS
20-
pip install git+https://github.com/fsschneider/DeepOBS.git
21-
22-
### Download the data
23-
deepobs_prepare_data.sh
24-
25-
This will automatically download, sort and prepare all the datasets (except ImageNet). It can take a while, as it will download roughly 1 GB.
26-
(If you already have the data, you could skip this step and always tell Deep OBS where the data is instead.)
27-
The data is now in a folder called 'data_deepobs'.
28-
29-
You are now ready to run different optimizers on different test problems, you can try for example
30-
31-
deepobs_run_sgd.py mnist.mnist_mlp --num_epochs=2 --lr=1e-1 --bs=128 --nologs
3+
![DeepOBS](docs/deepobs_banner.png "DeepOBS")
324

33-
to run SGD on a simple multi-layer perceptron (with a learning rate of 1e-1 and a batch size of 128 for 4 epochs without keeping logs).
5+
**DeepOBS** is a benchmarking suite that drastically simplifies, automates and
6+
improves the evaluation of deep learning optimizers.
347

35-
Of course, the real value of a benchmark lies in evaluating new optimizers:
8+
It can evaluate the performance of new optimizers on a variety of
9+
**real-world test problems** and automatically compare them with
10+
**realistic baselines**.
3611

37-
### Download and edit a run script
38-
You can download a template run script from there
12+
DeepOBS automates several steps when benchmarking deep learning optimizers:
3913

40-
https://github.com/fsschneider/DeepOBS/blob/master/scripts/deepobs_run_sgd.py
14+
- Downloading and preparing data sets.
15+
- Setting up test problems consisting of contemporary data sets and realistic
16+
deep learning architectures.
17+
- Running the optimizers on multiple test problems and logging relevant
18+
metrics.
19+
- Reporting and visualization the results of the optimizer benchmark.
4120

42-
Now you have a deepobs_run_script.py script in your folder. In order to run your optimizer, you need to change a few things in this script.
43-
The script takes take of the training, evaluation and logging.
44-
Let's assume that we want to benchmark the RMSProp optimizer. Then we only have to change Line 129 from
21+
![DeepOBS Output](docs/deepobs.jpg "DeepOBS_output")
4522

46-
opt = tf.train.GradientDescentOptimizer(lr)
23+
The code for the current implementation working with **TensorFlow** can be found
24+
on [Github](https://github.com/fsschneider/DeepOBS).
4725

48-
to
26+
The full documentation is available on readthedocs:
27+
https://deepobs.readthedocs.io/
4928

50-
opt = tf.train.RMSPropOptimizer(lr)
51-
52-
Usually the hyperparameters of the optimizers need to be included as well, but for now let's only take the learning rate as a hyperparameter for RMSProp (and if you want change all the 'sgd's in the comments to 'rmsprop'). Let's name this run script now deepobs_run_rmsprop.py
53-
54-
### Run your optimizer
55-
You can now run your optimizer on a test problem. Let's try it on a noisy quadratic problem:
56-
57-
python deepobs_run_rmsprop.py quadratic.noisy_quadratic --num_epochs=100 --lr=1e-1 --bs=128 --pickle --run_name=RMSProp_1e-1/
58-
59-
(we can repeat this a couple of times with different random seeds. This way, we will get a measure of uncertainty in the benchmark plots)
60-
61-
python deepobs_run_rmsprop.py quadratic.noisy_quadratic --num_epochs=100 --lr=1e-1 --bs=128 --pickle --run_name=RMSProp_1e-1/ --random_seed=43
62-
python deepobs_run_rmsprop.py quadratic.noisy_quadratic --num_epochs=100 --lr=1e-1 --bs=128 --pickle --run_name=RMSProp_1e-1/ --random_seed=44
63-
64-
You can monitor the training in real-time using Tensorboard
65-
66-
tensorboard --logdir=results
67-
68-
For this example, we will run the above code again, but with a different learning rate. We will call this "second optimizer" RMRProp_1e-2
69-
70-
python deepobs_run_rmsprop.py quadratic.noisy_quadratic --num_epochs=100 --lr=1e-2 --bs=128 --pickle --run_name=RMSProp_1e-2/
71-
python deepobs_run_rmsprop.py quadratic.noisy_quadratic --num_epochs=100 --lr=1e-2 --bs=128 --pickle --run_name=RMSProp_1e-2/ --random_seed=43
72-
python deepobs_run_rmsprop.py quadratic.noisy_quadratic --num_epochs=100 --lr=1e-2 --bs=128 --pickle --run_name=RMSProp_1e-2/ --random_seed=44
73-
74-
If you want to you can quickly run both optimizers on another problem
75-
76-
python deepobs_run_rmsprop.py mnist.mnist_mlp --num_epochs=5 --lr=1e-1 --bs=128 --pickle --run_name=RMSProp_1e-1/
77-
python deepobs_run_rmsprop.py mnist.mnist_mlp --num_epochs=5 --lr=1e-1 --bs=128 --pickle --run_name=RMSProp_1e-1/ --random_seed=43
78-
python deepobs_run_rmsprop.py mnist.mnist_mlp --num_epochs=5 --lr=1e-1 --bs=128 --pickle --run_name=RMSProp_1e-1/ --random_seed=44
29+
The paper describing DeepOBS has been accepted for ICLR 2019 and can be found
30+
here:
31+
https://openreview.net/forum?id=rJg6ssC5Y7
7932

80-
python deepobs_run_rmsprop.py mnist.mnist_mlp --num_epochs=5 --lr=1e-2 --bs=128 --pickle --run_name=RMSProp_1e-2/
81-
python deepobs_run_rmsprop.py mnist.mnist_mlp --num_epochs=5 --lr=1e-2 --bs=128 --pickle --run_name=RMSProp_1e-2/ --random_seed=43
82-
python deepobs_run_rmsprop.py mnist.mnist_mlp --num_epochs=5 --lr=1e-2 --bs=128 --pickle --run_name=RMSProp_1e-2/ --random_seed=44
33+
We are actively working on a **PyTorch** version and will be releasing it in the
34+
next months. In the meantime, PyTorch users can still use parts of DeepOBS such
35+
as the data preprocessing scripts or the visualization features.
8336

8437

85-
### Plot Results
86-
Now we can plot the results of those two "new" optimizers "RMSProp_1e-1" and "RMSProp_1e-2". Since the performance is always relative, we automatically plot the performance against the most popular optimizers (SGD, Momentum, Adam) with the best settings we found after tuning their hyperparameters. Try out:
38+
## Installation
8739

88-
deepobs_plot_results.py --results_dir=results --log
40+
pip install git+https://github.com/fsschneider/DeepOBS.git
8941

90-
which shows you the learning curves (loss and accuracy for both test and train dataset, but in the case of optimizing a quadratic, there is no accuracy) on a logarithmic plot.
91-
Additionally it will print out a table summarizing the performances over all test problems (here we only have one or two).
92-
If you add the option --saveto=save_dir the plots and a color coded table are saved as .png and ready-to-include .tex-files!
42+
Note, that the installation process can take a while as it will also
43+
automatically download all baseline results.
9344

94-
### Estimate runtime overhead
95-
You can estimate the runtime overhead of the new optimizers compared to SGD like this:
45+
We tested the package with Python 3.6 and TensorFlow version 1.12. Other
46+
versions of Python and TensorFlow (>= 1.4.0) might work, and we plan to expand
47+
compatibility in the future.
9648

97-
deepobs_estimate_runtime.py deepobs_run_rmsprop.py --optimizer_arguments=--lr=1e-2
98-
It will return an estimate of the overhead of the new optimizer compared to SGD. In our case it should be quite close to 1.0, as RMSProp costs roughly the same as SGD.
49+
Further tutorials and a suggested protocol for benchmarking deep learning
50+
optimizers can be found on https://deepobs.readthedocs.io/

‎__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# -*- coding: utf-8 -*-
2+
3+
from . import deepobs

‎deepobs/__init__.py‎

Lines changed: 4 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,5 @@
1-
import cifar100
2-
import cifar10
3-
import mnist
4-
import fmnist
5-
import tolstoi
6-
import imagenet
7-
import svhn
8-
import two_d
9-
import quadratic
1+
# -*- coding: utf-8 -*-
102

11-
import dataset_utils
12-
import run_utils
13-
import plot_utils
3+
from . import tensorflow
4+
from . import analyzer
5+
from . import scripts

‎deepobs/analyzer/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
from . import analyze_utils
2+
from . import analyze

‎deepobs/analyzer/analyze.py‎

Lines changed: 191 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,191 @@
1+
#!/usr/bin/env python
2+
3+
from __future__ import print_function
4+
5+
import matplotlib.pyplot as plt
6+
import seaborn as sns
7+
8+
import deepobs
9+
10+
sns.set()
11+
sns.set_style("whitegrid", {
12+
'axes.grid': False,
13+
'axes.spines.top': False,
14+
'axes.spines.right': False
15+
})
16+
17+
18+
def get_best_run(folder_pars):
19+
print("Get best run\n\n")
20+
for _, testprob in folder_pars.testproblems.items():
21+
print("***********************")
22+
print("Analyzing", testprob.name)
23+
print("***********************")
24+
for _, opt in testprob.optimizers.items():
25+
# print("Analyzing", opt.name)
26+
print("Checked", opt.num_settings, "settings for", opt.name,
27+
"and found the following")
28+
setting_final = opt.get_best_setting_final()
29+
setting_best = opt.get_best_setting_best()
30+
print("Best Setting (Final Value)", setting_final.name,
31+
"with final performance of",
32+
setting_final.aggregate.final_value)
33+
print("Best Setting (Best Value)", setting_best.name,
34+
"with best performance of",
35+
setting_best.aggregate.best_value)
36+
37+
38+
def plot_lr_sensitivity(folder_pars, baseline_pars=None, mode='final'):
39+
print("Plot learning rate sensitivity plot")
40+
fig, axis = plt.subplots(2, 4, figsize=(35, 4))
41+
42+
ax_row = 0
43+
for testprob in [
44+
"quadratic_deep", "mnist_vae", "fmnist_2c2d", "cifar10_3c3d"
45+
]:
46+
if testprob in folder_pars.testproblems:
47+
for _, opt in folder_pars.testproblems[testprob].optimizers.items(
48+
):
49+
opt.plot_lr_sensitivity(axis[0][ax_row], mode=mode)
50+
ax_row += 1
51+
if baseline_pars is not None:
52+
ax_row = 0
53+
for testprob in [
54+
"quadratic_deep", "mnist_vae", "fmnist_2c2d", "cifar10_3c3d"
55+
]:
56+
if testprob in baseline_pars.testproblems:
57+
for _, opt in baseline_pars.testproblems[
58+
testprob].optimizers.items():
59+
opt.plot_lr_sensitivity(axis[0][ax_row], mode=mode)
60+
ax_row += 1
61+
ax_row = 0
62+
for testprob in [
63+
"fmnist_vae", "cifar100_allcnnc", "svhn_wrn164", "tolstoi_char_rnn"
64+
]:
65+
if testprob in folder_pars.testproblems:
66+
for _, opt in folder_pars.testproblems[testprob].optimizers.items(
67+
):
68+
opt.plot_lr_sensitivity(axis[1][ax_row], mode=mode)
69+
ax_row += 1
70+
if baseline_pars is not None:
71+
ax_row = 0
72+
for testprob in [
73+
"fmnist_vae", "cifar100_allcnnc", "svhn_wrn164",
74+
"tolstoi_char_rnn"
75+
]:
76+
if testprob in baseline_pars.testproblems:
77+
for _, opt in baseline_pars.testproblems[
78+
testprob].optimizers.items():
79+
opt.plot_lr_sensitivity(axis[1][ax_row], mode=mode)
80+
ax_row += 1
81+
82+
fig, axis = deepobs.analyzer.analyze_utils.beautify_lr_sensitivity(
83+
fig, axis)
84+
deepobs.analyzer.analyze_utils.texify_lr_sensitivity(fig, axis)
85+
plt.show()
86+
87+
88+
def plot_performance(folder_pars, baseline_pars=None, mode="most"):
89+
# Small Benchmark
90+
fig, axis = plt.subplots(4, 4, sharex='col', figsize=(25, 8))
91+
92+
ax_col = 0
93+
for testprob in [
94+
"quadratic_deep", "mnist_vae", "fmnist_2c2d", "cifar10_3c3d"
95+
]:
96+
if testprob in folder_pars.testproblems:
97+
for _, opt in folder_pars.testproblems[testprob].optimizers.items(
98+
):
99+
opt.plot_performance(axis[:, ax_col], mode=mode)
100+
ax_col += 1
101+
if baseline_pars is not None:
102+
ax_col = 0
103+
for testprob in [
104+
"quadratic_deep", "mnist_vae", "fmnist_2c2d", "cifar10_3c3d"
105+
]:
106+
if testprob in baseline_pars.testproblems:
107+
for _, opt in baseline_pars.testproblems[
108+
testprob].optimizers.items():
109+
opt.plot_performance(axis[:, ax_col], mode='most')
110+
ax_col += 1
111+
fig, axis = deepobs.analyzer.analyze_utils.beautify_plot_performance(
112+
fig, axis, folder_pars, "small")
113+
deepobs.analyzer.analyze_utils.texify_plot_performance(fig, axis, "small")
114+
plt.show()
115+
116+
# Large Benchmark
117+
fig, axis = plt.subplots(4, 4, sharex='col', figsize=(25, 8))
118+
119+
ax_col = 0
120+
for testprob in [
121+
"fmnist_vae", "cifar100_allcnnc", "svhn_wrn164", "tolstoi_char_rnn"
122+
]:
123+
if testprob in folder_pars.testproblems:
124+
for _, opt in folder_pars.testproblems[testprob].optimizers.items():
125+
opt.plot_performance(axis[:, ax_col], mode=mode)
126+
ax_col += 1
127+
if baseline_pars is not None:
128+
ax_col = 0
129+
for testprob in [
130+
"fmnist_vae", "cifar100_allcnnc", "svhn_wrn164",
131+
"tolstoi_char_rnn"
132+
]:
133+
if testprob in baseline_pars.testproblems:
134+
for _, opt in baseline_pars.testproblems[
135+
testprob].optimizers.items():
136+
opt.plot_performance(axis[:, ax_col], mode='most')
137+
ax_col += 1
138+
fig, axis = deepobs.analyzer.analyze_utils.beautify_plot_performance(
139+
fig, axis, folder_pars, "large")
140+
deepobs.analyzer.analyze_utils.texify_plot_performance(fig, axis, "large")
141+
plt.show()
142+
143+
144+
def plot_table(folder_pars, baseline_pars=None):
145+
print("Plot overall performance table")
146+
147+
bm_table_small = dict()
148+
for testprob in [
149+
"quadratic_deep", "mnist_vae", "fmnist_2c2d", "cifar10_3c3d"
150+
]:
151+
bm_table_small[testprob] = dict()
152+
bm_table_small[testprob]['Performance'] = dict()
153+
bm_table_small[testprob]['Speed'] = dict()
154+
bm_table_small[testprob]['Tuneability'] = dict()
155+
if testprob in folder_pars.testproblems:
156+
for _, opt in folder_pars.testproblems[testprob].optimizers.items():
157+
bm_table_small[testprob] = opt.get_bm_table(
158+
bm_table_small[testprob])
159+
if baseline_pars is not None:
160+
if testprob in baseline_pars.testproblems:
161+
for _, opt in baseline_pars.testproblems[
162+
testprob].optimizers.items():
163+
bm_table_small[testprob] = opt.get_bm_table(
164+
bm_table_small[testprob])
165+
bm_table_small_pd = deepobs.analyzer.analyze_utils.beautify_plot_table(
166+
bm_table_small)
167+
deepobs.analyzer.analyze_utils.texify_plot_table(bm_table_small_pd,
168+
"small")
169+
170+
bm_table_large = dict()
171+
for testprob in [
172+
"fmnist_vae", "cifar100_allcnnc", "svhn_wrn164", "tolstoi_char_rnn"
173+
]:
174+
bm_table_large[testprob] = dict()
175+
bm_table_large[testprob]['Performance'] = dict()
176+
bm_table_large[testprob]['Speed'] = dict()
177+
bm_table_large[testprob]['Tuneability'] = dict()
178+
if testprob in folder_pars.testproblems:
179+
for _, opt in folder_pars.testproblems[testprob].optimizers.items():
180+
bm_table_large[testprob] = opt.get_bm_table(
181+
bm_table_large[testprob])
182+
if baseline_pars is not None:
183+
if testprob in baseline_pars.testproblems:
184+
for _, opt in baseline_pars.testproblems[
185+
testprob].optimizers.items():
186+
bm_table_large[testprob] = opt.get_bm_table(
187+
bm_table_large[testprob])
188+
bm_table_large_pd = deepobs.analyzer.analyze_utils.beautify_plot_table(
189+
bm_table_large)
190+
deepobs.analyzer.analyze_utils.texify_plot_table(bm_table_large_pd,
191+
"large")

0 commit comments

Comments
 (0)