Skip to content

Commit 30a34b0

Browse files
author
Github Actions
committed
Ravin Kohli: Reproducibility in cocktail (#204)
1 parent abf4a2d commit 30a34b0

19 files changed

+165
-179
lines changed

‎refactor_development_regularization_cocktails/_downloads/2f0f54a9831653ac5c12ee0e06324a62/basics_tabular_jupyter.zip

0 Bytes
Binary file not shown.

‎refactor_development_regularization_cocktails/_downloads/8ef6602cf8ed40221edd89244fb031af/basics_tabular_python.zip

0 Bytes
Binary file not shown.

‎refactor_development_regularization_cocktails/_downloads/b38295cf56f02b1c4547385a8f389f90/advanced_tabular_python.zip

0 Bytes
Binary file not shown.

‎refactor_development_regularization_cocktails/_downloads/df038041811db9bd567de83692a8f994/advanced_tabular_jupyter.zip

0 Bytes
Binary file not shown.

‎refactor_development_regularization_cocktails/_sources/advanced_tabular/example_custom_configuration_space.rst.txt

Lines changed: 27 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ the search. Currently, there are two changes that can be made to the space:-
4646

4747
.. code-block:: none
4848
49-
<smac.runhistory.runhistory.RunHistory object at 0x7f35a7c0b940> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
49+
<smac.runhistory.runhistory.RunHistory object at 0x7f69d7b96f10> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
5050
data_loader:batch_size, Value: 32
5151
encoder:__choice__, Value: 'OneHotEncoder'
5252
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -87,7 +87,7 @@ the search. Currently, there are two changes that can be made to the space:-
8787
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
8888
trainer:StandardTrainer:weighted_loss, Constant: 1
8989
trainer:__choice__, Value: 'StandardTrainer'
90-
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.00197601318359375, budget=0), TrajEntry(train_perf=0.16959064327485385, incumbent_id=1, incumbent=Configuration:
90+
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0021958351135253906, budget=0), TrajEntry(train_perf=0.16959064327485385, incumbent_id=1, incumbent=Configuration:
9191
data_loader:batch_size, Value: 32
9292
encoder:__choice__, Value: 'OneHotEncoder'
9393
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -128,16 +128,19 @@ the search. Currently, there are two changes that can be made to the space:-
128128
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
129129
trainer:StandardTrainer:weighted_loss, Constant: 1
130130
trainer:__choice__, Value: 'StandardTrainer'
131-
, ta_runs=1, ta_time_used=12.787853240966797, wallclock_time=14.251850366592407, budget=5.555555555555555)]
132-
{'accuracy': 0.8728323699421965}
133-
| | Preprocessing | Estimator | Weight |
134-
|---:|:----------------|:---------------------|---------:|
135-
| 0 | None | ExtraTreesClassifier | 0.4 |
136-
| 1 | None | RFClassifier | 0.24 |
137-
| 2 | None | LGBMClassifier | 0.2 |
138-
| 3 | None | KNNClassifier | 0.12 |
139-
| 4 | None | CatBoostClassifier | 0.04 |
140-
<smac.runhistory.runhistory.RunHistory object at 0x7f35a7c0bfa0> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
131+
, ta_runs=1, ta_time_used=15.160608291625977, wallclock_time=16.63932466506958, budget=5.555555555555555)]
132+
{'accuracy': 0.8670520231213873}
133+
| | Preprocessing | Estimator | Weight |
134+
|---:|:------------------------------------------------------------------|:-------------------------------------------------------------|---------:|
135+
| 0 | None | CatBoostClassifier | 0.34 |
136+
| 1 | None | RFClassifier | 0.24 |
137+
| 2 | SimpleImputer,OneHotEncoder,NoScaler,PolynomialFeatures | no embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.14 |
138+
| 3 | None | ExtraTreesClassifier | 0.1 |
139+
| 4 | None | LGBMClassifier | 0.08 |
140+
| 5 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,MLPBackbone,FullyConnectedHead,nn.Sequential | 0.06 |
141+
| 6 | None | SVC | 0.02 |
142+
| 7 | None | KNNClassifier | 0.02 |
143+
<smac.runhistory.runhistory.RunHistory object at 0x7f69ed87a580> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
141144
data_loader:batch_size, Value: 32
142145
encoder:__choice__, Value: 'NoEncoder'
143146
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -176,7 +179,7 @@ the search. Currently, there are two changes that can be made to the space:-
176179
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
177180
trainer:StandardTrainer:weighted_loss, Constant: 1
178181
trainer:__choice__, Value: 'StandardTrainer'
179-
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.001708984375, budget=0), TrajEntry(train_perf=0.2222222222222222, incumbent_id=1, incumbent=Configuration:
182+
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0015273094177246094, budget=0), TrajEntry(train_perf=0.2222222222222222, incumbent_id=1, incumbent=Configuration:
180183
data_loader:batch_size, Value: 32
181184
encoder:__choice__, Value: 'NoEncoder'
182185
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -215,18 +218,17 @@ the search. Currently, there are two changes that can be made to the space:-
215218
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
216219
trainer:StandardTrainer:weighted_loss, Constant: 1
217220
trainer:__choice__, Value: 'StandardTrainer'
218-
, ta_runs=1, ta_time_used=8.091186046600342, wallclock_time=9.551165580749512, budget=5.555555555555555)]
221+
, ta_runs=1, ta_time_used=9.499428987503052, wallclock_time=10.939727783203125, budget=5.555555555555555)]
219222
{'accuracy': 0.8554913294797688}
220-
| | Preprocessing | Estimator | Weight |
221-
|---:|:----------------------------------------------------|:----------------------------------------------------------------|---------:|
222-
| 0 | SimpleImputer,NoEncoder,NoScaler,PolynomialFeatures | no embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.4 |
223-
| 1 | None | CatBoostClassifier | 0.16 |
224-
| 2 | SimpleImputer,NoEncoder,Normalizer,KernelPCA | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.14 |
225-
| 3 | None | LGBMClassifier | 0.1 |
226-
| 4 | None | ExtraTreesClassifier | 0.08 |
227-
| 5 | None | RFClassifier | 0.06 |
228-
| 6 | None | SVC | 0.04 |
229-
| 7 | SimpleImputer,NoEncoder,NoScaler,PolynomialFeatures | no embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
223+
| | Preprocessing | Estimator | Weight |
224+
|---:|:----------------------------------------------------|:-------------------------------------------------------------|---------:|
225+
| 0 | SimpleImputer,NoEncoder,NoScaler,PolynomialFeatures | no embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.7 |
226+
| 1 | None | RFClassifier | 0.1 |
227+
| 2 | None | CatBoostClassifier | 0.06 |
228+
| 3 | SimpleImputer,NoEncoder,NoScaler,PolynomialFeatures | no embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.04 |
229+
| 4 | None | LGBMClassifier | 0.04 |
230+
| 5 | None | ExtraTreesClassifier | 0.04 |
231+
| 6 | None | SVC | 0.02 |
230232
231233
232234
@@ -355,7 +357,7 @@ the search. Currently, there are two changes that can be made to the space:-
355357
356358
.. rst-class:: sphx-glr-timing
357359

358-
**Total running time of the script:** ( 11 minutes 40.879 seconds)
360+
**Total running time of the script:** ( 11 minutes 37.611 seconds)
359361

360362

361363
.. _sphx_glr_download_advanced_tabular_example_custom_configuration_space.py:

‎refactor_development_regularization_cocktails/_sources/advanced_tabular/example_resampling_strategy.rst.txt

Lines changed: 27 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ a 67% train size split.
3838

3939
.. code-block:: none
4040
41-
<smac.runhistory.runhistory.RunHistory object at 0x7f35a79806a0> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
41+
<smac.runhistory.runhistory.RunHistory object at 0x7f69eda5e670> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
4242
data_loader:batch_size, Value: 64
4343
encoder:__choice__, Value: 'OneHotEncoder'
4444
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -77,7 +77,7 @@ a 67% train size split.
7777
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
7878
trainer:StandardTrainer:weighted_loss, Constant: 1
7979
trainer:__choice__, Value: 'StandardTrainer'
80-
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0018994808197021484, budget=0), TrajEntry(train_perf=0.16959064327485385, incumbent_id=1, incumbent=Configuration:
80+
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0019121170043945312, budget=0), TrajEntry(train_perf=0.16959064327485385, incumbent_id=1, incumbent=Configuration:
8181
data_loader:batch_size, Value: 64
8282
encoder:__choice__, Value: 'OneHotEncoder'
8383
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -116,19 +116,18 @@ a 67% train size split.
116116
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
117117
trainer:StandardTrainer:weighted_loss, Constant: 1
118118
trainer:__choice__, Value: 'StandardTrainer'
119-
, ta_runs=1, ta_time_used=12.164043426513672, wallclock_time=13.653866529464722, budget=5.555555555555555)]
120-
{'accuracy': 0.8786127167630058}
119+
, ta_runs=1, ta_time_used=14.97559404373169, wallclock_time=16.456933736801147, budget=5.555555555555555)]
120+
{'accuracy': 0.8728323699421965}
121121
| | Preprocessing | Estimator | Weight |
122122
|---:|:------------------------------------------------------------------|:----------------------------------------------------------------|---------:|
123-
| 0 | SimpleImputer,OneHotEncoder,Normalizer,PolynomialFeatures | embedding,MLPBackbone,FullyConnectedHead,nn.Sequential | 0.18 |
124-
| 1 | None | LGBMClassifier | 0.18 |
125-
| 2 | None | RFClassifier | 0.18 |
126-
| 3 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.16 |
127-
| 4 | None | CatBoostClassifier | 0.14 |
128-
| 5 | None | ExtraTreesClassifier | 0.1 |
129-
| 6 | None | SVC | 0.04 |
130-
| 7 | None | KNNClassifier | 0.02 |
131-
<smac.runhistory.runhistory.RunHistory object at 0x7f35c2192be0> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
123+
| 0 | None | LGBMClassifier | 0.22 |
124+
| 1 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.22 |
125+
| 2 | None | ExtraTreesClassifier | 0.2 |
126+
| 3 | None | RFClassifier | 0.16 |
127+
| 4 | None | SVC | 0.1 |
128+
| 5 | None | CatBoostClassifier | 0.08 |
129+
| 6 | None | KNNClassifier | 0.02 |
130+
<smac.runhistory.runhistory.RunHistory object at 0x7f69d7e87f10> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
132131
data_loader:batch_size, Value: 64
133132
encoder:__choice__, Value: 'OneHotEncoder'
134133
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -167,7 +166,7 @@ a 67% train size split.
167166
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
168167
trainer:StandardTrainer:weighted_loss, Constant: 1
169168
trainer:__choice__, Value: 'StandardTrainer'
170-
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0021066665649414062, budget=0), TrajEntry(train_perf=0.1623768221379145, incumbent_id=1, incumbent=Configuration:
169+
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0020830631256103516, budget=0), TrajEntry(train_perf=1.0, incumbent_id=1, incumbent=Configuration:
171170
data_loader:batch_size, Value: 64
172171
encoder:__choice__, Value: 'OneHotEncoder'
173172
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -206,18 +205,17 @@ a 67% train size split.
206205
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
207206
trainer:StandardTrainer:weighted_loss, Constant: 1
208207
trainer:__choice__, Value: 'StandardTrainer'
209-
, ta_runs=1, ta_time_used=30.45268940925598, wallclock_time=31.921412467956543, budget=5.555555555555555)]
210-
{'accuracy': 0.8554913294797688}
211-
| | Preprocessing | Estimator | Weight |
212-
|---:|:------------------------------------------------------------------|:----------------------------------------------------------------|---------:|
213-
| 0 | None | TabularClassifier | 0.22 |
214-
| 1 | None | TabularClassifier | 0.22 |
215-
| 2 | None | TabularClassifier | 0.22 |
216-
| 3 | None | TabularClassifier | 0.16 |
217-
| 4 | None | TabularClassifier | 0.1 |
218-
| 5 | None | TabularClassifier | 0.06 |
219-
| 6 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
220-
<smac.runhistory.runhistory.RunHistory object at 0x7f35a75718e0> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
208+
, ta_runs=1, ta_time_used=32.47600293159485, wallclock_time=33.94159984588623, budget=5.555555555555555)]
209+
{'accuracy': 0.8728323699421965}
210+
| | Preprocessing | Estimator | Weight |
211+
|---:|:----------------|:------------------|---------:|
212+
| 0 | None | TabularClassifier | 0.24 |
213+
| 1 | None | TabularClassifier | 0.24 |
214+
| 2 | None | TabularClassifier | 0.18 |
215+
| 3 | None | TabularClassifier | 0.16 |
216+
| 4 | None | TabularClassifier | 0.12 |
217+
| 5 | None | TabularClassifier | 0.06 |
218+
<smac.runhistory.runhistory.RunHistory object at 0x7f69edc521f0> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
221219
data_loader:batch_size, Value: 64
222220
encoder:__choice__, Value: 'OneHotEncoder'
223221
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -256,7 +254,7 @@ a 67% train size split.
256254
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
257255
trainer:StandardTrainer:weighted_loss, Constant: 1
258256
trainer:__choice__, Value: 'StandardTrainer'
259-
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0021975040435791016, budget=0), TrajEntry(train_perf=0.16374269005847952, incumbent_id=1, incumbent=Configuration:
257+
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0020017623901367188, budget=0), TrajEntry(train_perf=0.16374269005847952, incumbent_id=1, incumbent=Configuration:
260258
data_loader:batch_size, Value: 64
261259
encoder:__choice__, Value: 'OneHotEncoder'
262260
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -295,7 +293,7 @@ a 67% train size split.
295293
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
296294
trainer:StandardTrainer:weighted_loss, Constant: 1
297295
trainer:__choice__, Value: 'StandardTrainer'
298-
, ta_runs=1, ta_time_used=12.529094219207764, wallclock_time=14.05341625213623, budget=5.555555555555555)]
296+
, ta_runs=1, ta_time_used=15.390185832977295, wallclock_time=16.844798803329468, budget=5.555555555555555)]
299297
{'accuracy': 0.884393063583815}
300298
| | Preprocessing | Estimator | Weight |
301299
|---:|:----------------|:---------------------|---------:|
@@ -466,7 +464,7 @@ a 67% train size split.
466464
467465
.. rst-class:: sphx-glr-timing
468466

469-
**Total running time of the script:** ( 9 minutes 58.270 seconds)
467+
**Total running time of the script:** ( 10 minutes 7.307 seconds)
470468

471469

472470
.. _sphx_glr_download_advanced_tabular_example_resampling_strategy.py:

0 commit comments

Comments
 (0)