Skip to content

Commit 884b458

Browse files
author
Github Actions
committed
nabenabe0928: [feat] Add an object that realizes the perf over time viz (#331)
1 parent f9659f4 commit 884b458

File tree

54 files changed

+1131
-208
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

54 files changed

+1131
-208
lines changed

‎development/_downloads/250b72c5b7995e4f210c161ca8b36741/example_plot_over_time.py

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
"""
2+
==============================
3+
Plot the Performance over Time
4+
==============================
5+
6+
Auto-Pytorch uses SMAC to fit individual machine learning algorithms
7+
and then ensembles them together using `Ensemble Selection
8+
<https://www.cs.cornell.edu/~caruana/ctp/ct.papers/caruana.icml04.icdm06long.pdf>`_.
9+
10+
The following examples shows how to plot both the performance
11+
of the individual models and their respective ensemble.
12+
13+
Additionally, as we are compatible with matplotlib,
14+
you can input any args or kwargs that are compatible with ax.plot.
15+
In the case when you would like to create multipanel visualization,
16+
please input plt.Axes obtained from matplotlib.pyplot.subplots.
17+
18+
"""
19+
import warnings
20+
21+
import numpy as np
22+
import pandas as pd
23+
24+
from sklearn import model_selection
25+
26+
import matplotlib.pyplot as plt
27+
28+
from autoPyTorch.api.tabular_classification import TabularClassificationTask
29+
from autoPyTorch.utils.results_visualizer import PlotSettingParams
30+
31+
32+
warnings.simplefilter(action='ignore', category=UserWarning)
33+
warnings.simplefilter(action='ignore', category=FutureWarning)
34+
35+
36+
############################################################################
37+
# Task Definition
38+
# ===============
39+
n_samples, dim = 100, 2
40+
X = np.random.random((n_samples, dim)) * 2 - 1
41+
y = ((X ** 2).sum(axis=-1) < 2 / np.pi).astype(np.int32)
42+
print(y)
43+
44+
X, y = pd.DataFrame(X), pd.DataFrame(y)
45+
X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y)
46+
47+
############################################################################
48+
# API Instantiation and Searching
49+
# ===============================
50+
api = TabularClassificationTask(seed=42)
51+
52+
api.search(X_train=X_train, y_train=y_train, X_test=X_test, y_test=y_test,
53+
optimize_metric='accuracy', total_walltime_limit=120, func_eval_time_limit_secs=10)
54+
55+
############################################################################
56+
# Create Setting Parameters Object
57+
# ================================
58+
metric_name = 'accuracy'
59+
60+
params = PlotSettingParams(
61+
xscale='log',
62+
xlabel='Runtime',
63+
ylabel='Accuracy',
64+
title='Toy Example',
65+
show=False # If you would like to show, make it True
66+
)
67+
68+
############################################################################
69+
# Plot with the Specified Setting Parameters
70+
# ==========================================
71+
_, ax = plt.subplots()
72+
73+
api.plot_perf_over_time(
74+
ax=ax, # You do not have to provide.
75+
metric_name=metric_name,
76+
plot_setting_params=params,
77+
marker='*',
78+
markersize=10
79+
)
80+
81+
# plt.show() might cause issue depending on environments
82+
plt.savefig('example_plot_over_time.png')

‎development/_downloads/8410e1dea0ba7de106ac5e2a04cee84d/example_plot_over_time.ipynb

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "code",
5+
"execution_count": null,
6+
"metadata": {
7+
"collapsed": false
8+
},
9+
"outputs": [],
10+
"source": [
11+
"%matplotlib inline"
12+
]
13+
},
14+
{
15+
"cell_type": "markdown",
16+
"metadata": {},
17+
"source": [
18+
"\n# Plot the Performance over Time\n\nAuto-Pytorch uses SMAC to fit individual machine learning algorithms\nand then ensembles them together using `Ensemble Selection\n<https://www.cs.cornell.edu/~caruana/ctp/ct.papers/caruana.icml04.icdm06long.pdf>`_.\n\nThe following examples shows how to plot both the performance\nof the individual models and their respective ensemble.\n\nAdditionally, as we are compatible with matplotlib,\nyou can input any args or kwargs that are compatible with ax.plot.\nIn the case when you would like to create multipanel visualization,\nplease input plt.Axes obtained from matplotlib.pyplot.subplots.\n"
19+
]
20+
},
21+
{
22+
"cell_type": "code",
23+
"execution_count": null,
24+
"metadata": {
25+
"collapsed": false
26+
},
27+
"outputs": [],
28+
"source": [
29+
"import warnings\n\nimport numpy as np\nimport pandas as pd\n\nfrom sklearn import model_selection\n\nimport matplotlib.pyplot as plt\n\nfrom autoPyTorch.api.tabular_classification import TabularClassificationTask\nfrom autoPyTorch.utils.results_visualizer import PlotSettingParams\n\n\nwarnings.simplefilter(action='ignore', category=UserWarning)\nwarnings.simplefilter(action='ignore', category=FutureWarning)"
30+
]
31+
},
32+
{
33+
"cell_type": "markdown",
34+
"metadata": {},
35+
"source": [
36+
"## Task Definition\n\n"
37+
]
38+
},
39+
{
40+
"cell_type": "code",
41+
"execution_count": null,
42+
"metadata": {
43+
"collapsed": false
44+
},
45+
"outputs": [],
46+
"source": [
47+
"n_samples, dim = 100, 2\nX = np.random.random((n_samples, dim)) * 2 - 1\ny = ((X ** 2).sum(axis=-1) < 2 / np.pi).astype(np.int32)\nprint(y)\n\nX, y = pd.DataFrame(X), pd.DataFrame(y)\nX_train, X_test, y_train, y_test = model_selection.train_test_split(X, y)"
48+
]
49+
},
50+
{
51+
"cell_type": "markdown",
52+
"metadata": {},
53+
"source": [
54+
"## API Instantiation and Searching\n\n"
55+
]
56+
},
57+
{
58+
"cell_type": "code",
59+
"execution_count": null,
60+
"metadata": {
61+
"collapsed": false
62+
},
63+
"outputs": [],
64+
"source": [
65+
"api = TabularClassificationTask(seed=42)\n\napi.search(X_train=X_train, y_train=y_train, X_test=X_test, y_test=y_test,\n optimize_metric='accuracy', total_walltime_limit=120, func_eval_time_limit_secs=10)"
66+
]
67+
},
68+
{
69+
"cell_type": "markdown",
70+
"metadata": {},
71+
"source": [
72+
"## Create Setting Parameters Object\n\n"
73+
]
74+
},
75+
{
76+
"cell_type": "code",
77+
"execution_count": null,
78+
"metadata": {
79+
"collapsed": false
80+
},
81+
"outputs": [],
82+
"source": [
83+
"metric_name = 'accuracy'\n\nparams = PlotSettingParams(\n xscale='log',\n xlabel='Runtime',\n ylabel='Accuracy',\n title='Toy Example',\n show=False # If you would like to show, make it True\n)"
84+
]
85+
},
86+
{
87+
"cell_type": "markdown",
88+
"metadata": {},
89+
"source": [
90+
"## Plot with the Specified Setting Parameters\n\n"
91+
]
92+
},
93+
{
94+
"cell_type": "code",
95+
"execution_count": null,
96+
"metadata": {
97+
"collapsed": false
98+
},
99+
"outputs": [],
100+
"source": [
101+
"_, ax = plt.subplots()\n\napi.plot_perf_over_time(\n ax=ax, # You do not have to provide.\n metric_name=metric_name,\n plot_setting_params=params,\n marker='*',\n markersize=10\n)\n\n# plt.show() might cause issue depending on environments\nplt.savefig('example_plot_over_time.png')"
102+
]
103+
}
104+
],
105+
"metadata": {
106+
"kernelspec": {
107+
"display_name": "Python 3",
108+
"language": "python",
109+
"name": "python3"
110+
},
111+
"language_info": {
112+
"codemirror_mode": {
113+
"name": "ipython",
114+
"version": 3
115+
},
116+
"file_extension": ".py",
117+
"mimetype": "text/x-python",
118+
"name": "python",
119+
"nbconvert_exporter": "python",
120+
"pygments_lexer": "ipython3",
121+
"version": "3.8.12"
122+
}
123+
},
124+
"nbformat": 4,
125+
"nbformat_minor": 0
126+
}

‎development/_downloads/bc82bea3a5dd7bdba60b65220891d9e5/examples_python.zip

2.7 KB
Binary file not shown.

‎development/_downloads/fb625db3c50d423b1b7881136ffdeec8/examples_jupyter.zip

4.36 KB
Binary file not shown.

‎development/_images/sphx_glr_example_plot_over_time_001.png

54.1 KB
Loading

‎development/_images/sphx_glr_example_plot_over_time_thumb.png

52.7 KB
Loading

‎development/_images/sphx_glr_example_visualization_001.png

-32 Bytes
Loading

‎development/_images/sphx_glr_example_visualization_002.png

456 Bytes
Loading

‎development/_images/sphx_glr_example_visualization_thumb.png

-958 Bytes
Loading

‎development/_modules/autoPyTorch/api/tabular_classification.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -486,7 +486,7 @@ <h1>Source code for autoPyTorch.api.tabular_classification</h1><div class="highl
486486
</p>
487487
<p>
488488
&copy; Copyright 2014-2021, Machine Learning Professorship Freiburg.<br/>
489-
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 4.3.0.<br/>
489+
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 4.3.1.<br/>
490490
</p>
491491
</div>
492492
</footer>

‎development/_modules/autoPyTorch/api/tabular_regression.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -463,7 +463,7 @@ <h1>Source code for autoPyTorch.api.tabular_regression</h1><div class="highlight
463463
</p>
464464
<p>
465465
&copy; Copyright 2014-2021, Machine Learning Professorship Freiburg.<br/>
466-
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 4.3.0.<br/>
466+
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 4.3.1.<br/>
467467
</p>
468468
</div>
469469
</footer>

‎development/_modules/autoPyTorch/pipeline/components/base_choice.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -434,7 +434,7 @@ <h1>Source code for autoPyTorch.pipeline.components.base_choice</h1><div class="
434434
</p>
435435
<p>
436436
&copy; Copyright 2014-2021, Machine Learning Professorship Freiburg.<br/>
437-
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 4.3.0.<br/>
437+
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 4.3.1.<br/>
438438
</p>
439439
</div>
440440
</footer>

‎development/_modules/autoPyTorch/pipeline/components/base_component.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -442,7 +442,7 @@ <h1>Source code for autoPyTorch.pipeline.components.base_component</h1><div clas
442442
</p>
443443
<p>
444444
&copy; Copyright 2014-2021, Machine Learning Professorship Freiburg.<br/>
445-
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 4.3.0.<br/>
445+
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 4.3.1.<br/>
446446
</p>
447447
</div>
448448
</footer>

‎development/_modules/autoPyTorch/pipeline/tabular_classification.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -502,7 +502,7 @@ <h1>Source code for autoPyTorch.pipeline.tabular_classification</h1><div class="
502502
</p>
503503
<p>
504504
&copy; Copyright 2014-2021, Machine Learning Professorship Freiburg.<br/>
505-
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 4.3.0.<br/>
505+
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 4.3.1.<br/>
506506
</p>
507507
</div>
508508
</footer>

‎development/_modules/autoPyTorch/pipeline/tabular_regression.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -452,7 +452,7 @@ <h1>Source code for autoPyTorch.pipeline.tabular_regression</h1><div class="high
452452
</p>
453453
<p>
454454
&copy; Copyright 2014-2021, Machine Learning Professorship Freiburg.<br/>
455-
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 4.3.0.<br/>
455+
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 4.3.1.<br/>
456456
</p>
457457
</div>
458458
</footer>

‎development/_modules/autoPyTorch/pipeline/traditional_tabular_classification.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -399,7 +399,7 @@ <h1>Source code for autoPyTorch.pipeline.traditional_tabular_classification</h1>
399399
</p>
400400
<p>
401401
&copy; Copyright 2014-2021, Machine Learning Professorship Freiburg.<br/>
402-
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 4.3.0.<br/>
402+
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 4.3.1.<br/>
403403
</p>
404404
</div>
405405
</footer>

‎development/_modules/autoPyTorch/pipeline/traditional_tabular_regression.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -355,7 +355,7 @@ <h1>Source code for autoPyTorch.pipeline.traditional_tabular_regression</h1><div
355355
</p>
356356
<p>
357357
&copy; Copyright 2014-2021, Machine Learning Professorship Freiburg.<br/>
358-
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 4.3.0.<br/>
358+
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 4.3.1.<br/>
359359
</p>
360360
</div>
361361
</footer>

‎development/_modules/index.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -136,7 +136,7 @@ <h1>All modules for which code is available</h1>
136136
</p>
137137
<p>
138138
&copy; Copyright 2014-2021, Machine Learning Professorship Freiburg.<br/>
139-
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 4.3.0.<br/>
139+
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 4.3.1.<br/>
140140
</p>
141141
</div>
142142
</footer>

‎development/_sources/examples/20_basics/example_image_classification.rst.txt

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -85,22 +85,22 @@ Image Classification
8585
Pipeline Random Config:
8686
________________________________________
8787
Configuration:
88-
image_augmenter:GaussianBlur:sigma_min, Value: 1.6194687445458993
89-
image_augmenter:GaussianBlur:sigma_offset, Value: 0.15419645415994476
90-
image_augmenter:GaussianBlur:use_augmenter, Value: True
91-
image_augmenter:GaussianNoise:use_augmenter, Value: False
88+
image_augmenter:GaussianBlur:use_augmenter, Value: False
89+
image_augmenter:GaussianNoise:sigma_offset, Value: 0.34378220529402603
90+
image_augmenter:GaussianNoise:use_augmenter, Value: True
9291
image_augmenter:RandomAffine:use_augmenter, Value: False
93-
image_augmenter:RandomCutout:use_augmenter, Value: False
92+
image_augmenter:RandomCutout:p, Value: 0.6073713136709832
93+
image_augmenter:RandomCutout:use_augmenter, Value: True
9494
image_augmenter:Resize:use_augmenter, Value: True
95-
image_augmenter:ZeroPadAndCrop:percent, Value: 0.09356660599930805
96-
normalizer:__choice__, Value: 'ImageNormalizer'
95+
image_augmenter:ZeroPadAndCrop:percent, Value: 0.4430360788212591
96+
normalizer:__choice__, Value: 'NoNormalizer'
9797

9898
Fitting the pipeline...
9999
________________________________________
100100
ImageClassificationPipeline
101101
________________________________________
102102
0-) normalizer:
103-
ImageNormalizer
103+
NoNormalizer
104104

105105
1-) preprocessing:
106106
EarlyPreprocessing
@@ -172,7 +172,7 @@ Image Classification
172172
173173
.. rst-class:: sphx-glr-timing
174174

175-
**Total running time of the script:** ( 0 minutes 6.174 seconds)
175+
**Total running time of the script:** ( 0 minutes 6.062 seconds)
176176

177177

178178
.. _sphx_glr_download_examples_20_basics_example_image_classification.py:

‎development/_sources/examples/20_basics/example_tabular_classification.rst.txt

Lines changed: 16 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -134,7 +134,7 @@ Search for an ensemble of machine learning algorithms
134134
.. code-block:: none
135135
136136
137-
<autoPyTorch.api.tabular_classification.TabularClassificationTask object at 0x7f4974e3e820>
137+
<autoPyTorch.api.tabular_classification.TabularClassificationTask object at 0x7fc43748c250>
138138
139139
140140
@@ -165,20 +165,23 @@ Print the final ensemble performance
165165

166166
.. code-block:: none
167167
168-
{'accuracy': 0.838150289017341}
168+
{'accuracy': 0.8554913294797688}
169169
| | Preprocessing | Estimator | Weight |
170170
|---:|:------------------------------------------------------------------|:----------------------------------------------------------------|---------:|
171-
| 0 | SimpleImputer,OneHotEncoder,Normalizer,KernelPCA | embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.2 |
171+
| 0 | SimpleImputer,OneHotEncoder,Normalizer,KernelPCA | embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.18 |
172172
| 1 | None | KNNLearner | 0.16 |
173-
| 2 | None | CBLearner | 0.14 |
174-
| 3 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.12 |
175-
| 4 | SimpleImputer,OneHotEncoder,Normalizer,PowerTransformer | embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.08 |
176-
| 5 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.08 |
177-
| 6 | SimpleImputer,OneHotEncoder,MinMaxScaler,PowerTransformer | embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.06 |
178-
| 7 | SimpleImputer,NoEncoder,Normalizer,Nystroem | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.06 |
179-
| 8 | None | SVMLearner | 0.04 |
180-
| 9 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.04 |
181-
| 10 | SimpleImputer,OneHotEncoder,MinMaxScaler,TruncSVD | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
173+
| 2 | None | SVMLearner | 0.12 |
174+
| 3 | None | CBLearner | 0.1 |
175+
| 4 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.08 |
176+
| 5 | SimpleImputer,OneHotEncoder,Normalizer,PowerTransformer | embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.06 |
177+
| 6 | None | RFLearner | 0.06 |
178+
| 7 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.06 |
179+
| 8 | SimpleImputer,OneHotEncoder,MinMaxScaler,PowerTransformer | embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.04 |
180+
| 9 | SimpleImputer,NoEncoder,Normalizer,Nystroem | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.04 |
181+
| 10 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.04 |
182+
| 11 | SimpleImputer,OneHotEncoder,Normalizer,TruncSVD | embedding,MLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
183+
| 12 | SimpleImputer,OneHotEncoder,MinMaxScaler,TruncSVD | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
184+
| 13 | None | ETLearner | 0.02 |
182185
autoPyTorch results:
183186
Dataset name: Australian
184187
Optimisation Metric: accuracy
@@ -196,7 +199,7 @@ Print the final ensemble performance
196199
197200
.. rst-class:: sphx-glr-timing
198201

199-
**Total running time of the script:** ( 5 minutes 31.629 seconds)
202+
**Total running time of the script:** ( 5 minutes 33.865 seconds)
200203

201204

202205
.. _sphx_glr_download_examples_20_basics_example_tabular_classification.py:

0 commit comments

Comments
 (0)