Skip to content

Commit ce79efc

Browse files
author
Github Actions
committed
Ravin Kohli: fix bug in adversarial trainer (#207)
1 parent 30a34b0 commit ce79efc

19 files changed

+261
-97
lines changed

‎refactor_development_regularization_cocktails/_downloads/2f0f54a9831653ac5c12ee0e06324a62/basics_tabular_jupyter.zip

0 Bytes
Binary file not shown.

‎refactor_development_regularization_cocktails/_downloads/8ef6602cf8ed40221edd89244fb031af/basics_tabular_python.zip

0 Bytes
Binary file not shown.

‎refactor_development_regularization_cocktails/_downloads/b38295cf56f02b1c4547385a8f389f90/advanced_tabular_python.zip

0 Bytes
Binary file not shown.

‎refactor_development_regularization_cocktails/_downloads/df038041811db9bd567de83692a8f994/advanced_tabular_jupyter.zip

0 Bytes
Binary file not shown.

‎refactor_development_regularization_cocktails/_sources/advanced_tabular/example_custom_configuration_space.rst.txt

Lines changed: 19 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ the search. Currently, there are two changes that can be made to the space:-
4646

4747
.. code-block:: none
4848
49-
<smac.runhistory.runhistory.RunHistory object at 0x7f69d7b96f10> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
49+
<smac.runhistory.runhistory.RunHistory object at 0x7fac3963a9d0> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
5050
data_loader:batch_size, Value: 32
5151
encoder:__choice__, Value: 'OneHotEncoder'
5252
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -87,7 +87,7 @@ the search. Currently, there are two changes that can be made to the space:-
8787
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
8888
trainer:StandardTrainer:weighted_loss, Constant: 1
8989
trainer:__choice__, Value: 'StandardTrainer'
90-
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0021958351135253906, budget=0), TrajEntry(train_perf=0.16959064327485385, incumbent_id=1, incumbent=Configuration:
90+
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0019087791442871094, budget=0), TrajEntry(train_perf=0.16959064327485385, incumbent_id=1, incumbent=Configuration:
9191
data_loader:batch_size, Value: 32
9292
encoder:__choice__, Value: 'OneHotEncoder'
9393
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -128,7 +128,7 @@ the search. Currently, there are two changes that can be made to the space:-
128128
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
129129
trainer:StandardTrainer:weighted_loss, Constant: 1
130130
trainer:__choice__, Value: 'StandardTrainer'
131-
, ta_runs=1, ta_time_used=15.160608291625977, wallclock_time=16.63932466506958, budget=5.555555555555555)]
131+
, ta_runs=1, ta_time_used=13.619905710220337, wallclock_time=15.071529388427734, budget=5.555555555555555)]
132132
{'accuracy': 0.8670520231213873}
133133
| | Preprocessing | Estimator | Weight |
134134
|---:|:------------------------------------------------------------------|:-------------------------------------------------------------|---------:|
@@ -140,7 +140,7 @@ the search. Currently, there are two changes that can be made to the space:-
140140
| 5 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,MLPBackbone,FullyConnectedHead,nn.Sequential | 0.06 |
141141
| 6 | None | SVC | 0.02 |
142142
| 7 | None | KNNClassifier | 0.02 |
143-
<smac.runhistory.runhistory.RunHistory object at 0x7f69ed87a580> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
143+
<smac.runhistory.runhistory.RunHistory object at 0x7fac28dcde80> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
144144
data_loader:batch_size, Value: 32
145145
encoder:__choice__, Value: 'NoEncoder'
146146
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -179,7 +179,7 @@ the search. Currently, there are two changes that can be made to the space:-
179179
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
180180
trainer:StandardTrainer:weighted_loss, Constant: 1
181181
trainer:__choice__, Value: 'StandardTrainer'
182-
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0015273094177246094, budget=0), TrajEntry(train_perf=0.2222222222222222, incumbent_id=1, incumbent=Configuration:
182+
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0019690990447998047, budget=0), TrajEntry(train_perf=0.2222222222222222, incumbent_id=1, incumbent=Configuration:
183183
data_loader:batch_size, Value: 32
184184
encoder:__choice__, Value: 'NoEncoder'
185185
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -218,17 +218,19 @@ the search. Currently, there are two changes that can be made to the space:-
218218
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
219219
trainer:StandardTrainer:weighted_loss, Constant: 1
220220
trainer:__choice__, Value: 'StandardTrainer'
221-
, ta_runs=1, ta_time_used=9.499428987503052, wallclock_time=10.939727783203125, budget=5.555555555555555)]
222-
{'accuracy': 0.8554913294797688}
223-
| | Preprocessing | Estimator | Weight |
224-
|---:|:----------------------------------------------------|:-------------------------------------------------------------|---------:|
225-
| 0 | SimpleImputer,NoEncoder,NoScaler,PolynomialFeatures | no embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.7 |
226-
| 1 | None | RFClassifier | 0.1 |
227-
| 2 | None | CatBoostClassifier | 0.06 |
228-
| 3 | SimpleImputer,NoEncoder,NoScaler,PolynomialFeatures | no embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.04 |
229-
| 4 | None | LGBMClassifier | 0.04 |
230-
| 5 | None | ExtraTreesClassifier | 0.04 |
231-
| 6 | None | SVC | 0.02 |
221+
, ta_runs=1, ta_time_used=8.54720139503479, wallclock_time=9.972944259643555, budget=5.555555555555555)]
222+
{'accuracy': 0.8670520231213873}
223+
| | Preprocessing | Estimator | Weight |
224+
|---:|:------------------------------------------------------------|:-------------------------------------------------------------------|---------:|
225+
| 0 | SimpleImputer,NoEncoder,MinMaxScaler,NoFeaturePreprocessing | no embedding,ShapedResNetBackbone,FullyConnectedHead,nn.Sequential | 0.3 |
226+
| 1 | SimpleImputer,NoEncoder,NoScaler,PolynomialFeatures | no embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.28 |
227+
| 2 | None | RFClassifier | 0.16 |
228+
| 3 | None | LGBMClassifier | 0.08 |
229+
| 4 | None | CatBoostClassifier | 0.06 |
230+
| 5 | None | SVC | 0.04 |
231+
| 6 | None | KNNClassifier | 0.04 |
232+
| 7 | SimpleImputer,NoEncoder,StandardScaler,PolynomialFeatures | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
233+
| 8 | None | ExtraTreesClassifier | 0.02 |
232234
233235
234236
@@ -357,7 +359,7 @@ the search. Currently, there are two changes that can be made to the space:-
357359
358360
.. rst-class:: sphx-glr-timing
359361

360-
**Total running time of the script:** ( 11 minutes 37.611 seconds)
362+
**Total running time of the script:** ( 11 minutes 29.301 seconds)
361363

362364

363365
.. _sphx_glr_download_advanced_tabular_example_custom_configuration_space.py:

‎refactor_development_regularization_cocktails/_sources/advanced_tabular/example_resampling_strategy.rst.txt

Lines changed: 88 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ a 67% train size split.
3838

3939
.. code-block:: none
4040
41-
<smac.runhistory.runhistory.RunHistory object at 0x7f69eda5e670> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
41+
<smac.runhistory.runhistory.RunHistory object at 0x7fac39191f10> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
4242
data_loader:batch_size, Value: 64
4343
encoder:__choice__, Value: 'OneHotEncoder'
4444
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -77,7 +77,7 @@ a 67% train size split.
7777
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
7878
trainer:StandardTrainer:weighted_loss, Constant: 1
7979
trainer:__choice__, Value: 'StandardTrainer'
80-
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0019121170043945312, budget=0), TrajEntry(train_perf=0.16959064327485385, incumbent_id=1, incumbent=Configuration:
80+
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0018260478973388672, budget=0), TrajEntry(train_perf=0.16959064327485385, incumbent_id=1, incumbent=Configuration:
8181
data_loader:batch_size, Value: 64
8282
encoder:__choice__, Value: 'OneHotEncoder'
8383
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -116,7 +116,7 @@ a 67% train size split.
116116
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
117117
trainer:StandardTrainer:weighted_loss, Constant: 1
118118
trainer:__choice__, Value: 'StandardTrainer'
119-
, ta_runs=1, ta_time_used=14.97559404373169, wallclock_time=16.456933736801147, budget=5.555555555555555)]
119+
, ta_runs=1, ta_time_used=13.256181716918945, wallclock_time=14.80335021018982, budget=5.555555555555555)]
120120
{'accuracy': 0.8728323699421965}
121121
| | Preprocessing | Estimator | Weight |
122122
|---:|:------------------------------------------------------------------|:----------------------------------------------------------------|---------:|
@@ -127,7 +127,7 @@ a 67% train size split.
127127
| 4 | None | SVC | 0.1 |
128128
| 5 | None | CatBoostClassifier | 0.08 |
129129
| 6 | None | KNNClassifier | 0.02 |
130-
<smac.runhistory.runhistory.RunHistory object at 0x7f69d7e87f10> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
130+
<smac.runhistory.runhistory.RunHistory object at 0x7fac38204f70> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
131131
data_loader:batch_size, Value: 64
132132
encoder:__choice__, Value: 'OneHotEncoder'
133133
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -166,7 +166,7 @@ a 67% train size split.
166166
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
167167
trainer:StandardTrainer:weighted_loss, Constant: 1
168168
trainer:__choice__, Value: 'StandardTrainer'
169-
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0020830631256103516, budget=0), TrajEntry(train_perf=1.0, incumbent_id=1, incumbent=Configuration:
169+
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.001847982406616211, budget=0), TrajEntry(train_perf=1.0, incumbent_id=1, incumbent=Configuration:
170170
data_loader:batch_size, Value: 64
171171
encoder:__choice__, Value: 'OneHotEncoder'
172172
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -205,7 +205,85 @@ a 67% train size split.
205205
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
206206
trainer:StandardTrainer:weighted_loss, Constant: 1
207207
trainer:__choice__, Value: 'StandardTrainer'
208-
, ta_runs=1, ta_time_used=32.47600293159485, wallclock_time=33.94159984588623, budget=5.555555555555555)]
208+
, ta_runs=1, ta_time_used=32.332130908966064, wallclock_time=33.77369141578674, budget=5.555555555555555), TrajEntry(train_perf=0.5245086861210053, incumbent_id=2, incumbent=Configuration:
209+
data_loader:batch_size, Value: 168
210+
encoder:__choice__, Value: 'NoEncoder'
211+
feature_preprocessor:PolynomialFeatures:degree, Value: 3
212+
feature_preprocessor:PolynomialFeatures:include_bias, Value: True
213+
feature_preprocessor:PolynomialFeatures:interaction_only, Value: False
214+
feature_preprocessor:__choice__, Value: 'PolynomialFeatures'
215+
imputer:categorical_strategy, Value: 'most_frequent'
216+
imputer:numerical_strategy, Value: 'median'
217+
lr_scheduler:CosineAnnealingWarmRestarts:n_restarts, Value: 3
218+
lr_scheduler:__choice__, Value: 'CosineAnnealingWarmRestarts'
219+
network_backbone:ResNetBackbone:activation, Value: 'relu'
220+
network_backbone:ResNetBackbone:blocks_per_group_0, Value: 2
221+
network_backbone:ResNetBackbone:blocks_per_group_1, Value: 1
222+
network_backbone:ResNetBackbone:blocks_per_group_10, Value: 2
223+
network_backbone:ResNetBackbone:blocks_per_group_11, Value: 4
224+
network_backbone:ResNetBackbone:blocks_per_group_2, Value: 3
225+
network_backbone:ResNetBackbone:blocks_per_group_3, Value: 2
226+
network_backbone:ResNetBackbone:blocks_per_group_4, Value: 4
227+
network_backbone:ResNetBackbone:blocks_per_group_5, Value: 2
228+
network_backbone:ResNetBackbone:blocks_per_group_6, Value: 2
229+
network_backbone:ResNetBackbone:blocks_per_group_7, Value: 3
230+
network_backbone:ResNetBackbone:blocks_per_group_8, Value: 2
231+
network_backbone:ResNetBackbone:blocks_per_group_9, Value: 1
232+
network_backbone:ResNetBackbone:dropout_0, Value: 0.20048655548352112
233+
network_backbone:ResNetBackbone:dropout_1, Value: 0.3704590136122547
234+
network_backbone:ResNetBackbone:dropout_10, Value: 0.47997613956763374
235+
network_backbone:ResNetBackbone:dropout_11, Value: 0.6028491783597432
236+
network_backbone:ResNetBackbone:dropout_2, Value: 0.6491115705749217
237+
network_backbone:ResNetBackbone:dropout_3, Value: 0.6970633645157825
238+
network_backbone:ResNetBackbone:dropout_4, Value: 0.5603777016112169
239+
network_backbone:ResNetBackbone:dropout_5, Value: 0.3178286700893052
240+
network_backbone:ResNetBackbone:dropout_6, Value: 0.5095780586533312
241+
network_backbone:ResNetBackbone:dropout_7, Value: 0.6914751560536525
242+
network_backbone:ResNetBackbone:dropout_8, Value: 0.15658944938766944
243+
network_backbone:ResNetBackbone:dropout_9, Value: 0.7576622371405439
244+
network_backbone:ResNetBackbone:max_shake_drop_probability, Value: 0.2033899238283674
245+
network_backbone:ResNetBackbone:multi_branch_choice, Value: 'shake-drop'
246+
network_backbone:ResNetBackbone:num_groups, Value: 11
247+
network_backbone:ResNetBackbone:num_units_0, Value: 433
248+
network_backbone:ResNetBackbone:num_units_1, Value: 24
249+
network_backbone:ResNetBackbone:num_units_10, Value: 136
250+
network_backbone:ResNetBackbone:num_units_11, Value: 637
251+
network_backbone:ResNetBackbone:num_units_2, Value: 229
252+
network_backbone:ResNetBackbone:num_units_3, Value: 71
253+
network_backbone:ResNetBackbone:num_units_4, Value: 148
254+
network_backbone:ResNetBackbone:num_units_5, Value: 96
255+
network_backbone:ResNetBackbone:num_units_6, Value: 258
256+
network_backbone:ResNetBackbone:num_units_7, Value: 15
257+
network_backbone:ResNetBackbone:num_units_8, Value: 119
258+
network_backbone:ResNetBackbone:num_units_9, Value: 50
259+
network_backbone:ResNetBackbone:use_batch_norm, Value: False
260+
network_backbone:ResNetBackbone:use_dropout, Value: True
261+
network_backbone:ResNetBackbone:use_skip_connection, Value: True
262+
network_backbone:__choice__, Value: 'ResNetBackbone'
263+
network_embedding:__choice__, Value: 'NoEmbedding'
264+
network_head:__choice__, Value: 'fully_connected'
265+
network_head:fully_connected:activation, Value: 'tanh'
266+
network_head:fully_connected:num_layers, Value: 3
267+
network_head:fully_connected:units_layer_1, Value: 438
268+
network_head:fully_connected:units_layer_2, Value: 311
269+
network_init:KaimingInit:bias_strategy, Value: 'Normal'
270+
network_init:__choice__, Value: 'KaimingInit'
271+
optimizer:AdamOptimizer:beta1, Value: 0.8511245386400297
272+
optimizer:AdamOptimizer:beta2, Value: 0.9786583401558273
273+
optimizer:AdamOptimizer:lr, Value: 0.00267654957157753
274+
optimizer:AdamOptimizer:use_weight_decay, Value: True
275+
optimizer:AdamOptimizer:weight_decay, Value: 3.572243966285694e-07
276+
optimizer:__choice__, Value: 'AdamOptimizer'
277+
scaler:Normalizer:norm, Value: 'mean_squared'
278+
scaler:__choice__, Value: 'Normalizer'
279+
trainer:MixUpTrainer:alpha, Value: 0.09333206992790188
280+
trainer:MixUpTrainer:se_lastk, Constant: 3
281+
trainer:MixUpTrainer:use_lookahead_optimizer, Value: False
282+
trainer:MixUpTrainer:use_snapshot_ensemble, Value: True
283+
trainer:MixUpTrainer:use_stochastic_weight_averaging, Value: False
284+
trainer:MixUpTrainer:weighted_loss, Constant: 1
285+
trainer:__choice__, Value: 'MixUpTrainer'
286+
, ta_runs=2, ta_time_used=63.04415440559387, wallclock_time=66.02159237861633, budget=5.555555555555555)]
209287
{'accuracy': 0.8728323699421965}
210288
| | Preprocessing | Estimator | Weight |
211289
|---:|:----------------|:------------------|---------:|
@@ -215,7 +293,7 @@ a 67% train size split.
215293
| 3 | None | TabularClassifier | 0.16 |
216294
| 4 | None | TabularClassifier | 0.12 |
217295
| 5 | None | TabularClassifier | 0.06 |
218-
<smac.runhistory.runhistory.RunHistory object at 0x7f69edc521f0> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
296+
<smac.runhistory.runhistory.RunHistory object at 0x7fac28372790> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
219297
data_loader:batch_size, Value: 64
220298
encoder:__choice__, Value: 'OneHotEncoder'
221299
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -254,7 +332,7 @@ a 67% train size split.
254332
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
255333
trainer:StandardTrainer:weighted_loss, Constant: 1
256334
trainer:__choice__, Value: 'StandardTrainer'
257-
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0020017623901367188, budget=0), TrajEntry(train_perf=0.16374269005847952, incumbent_id=1, incumbent=Configuration:
335+
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0016481876373291016, budget=0), TrajEntry(train_perf=0.16374269005847952, incumbent_id=1, incumbent=Configuration:
258336
data_loader:batch_size, Value: 64
259337
encoder:__choice__, Value: 'OneHotEncoder'
260338
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -293,7 +371,7 @@ a 67% train size split.
293371
trainer:StandardTrainer:use_stochastic_weight_averaging, Value: True
294372
trainer:StandardTrainer:weighted_loss, Constant: 1
295373
trainer:__choice__, Value: 'StandardTrainer'
296-
, ta_runs=1, ta_time_used=15.390185832977295, wallclock_time=16.844798803329468, budget=5.555555555555555)]
374+
, ta_runs=1, ta_time_used=13.66442322731018, wallclock_time=15.141267538070679, budget=5.555555555555555)]
297375
{'accuracy': 0.884393063583815}
298376
| | Preprocessing | Estimator | Weight |
299377
|---:|:----------------|:---------------------|---------:|
@@ -464,7 +542,7 @@ a 67% train size split.
464542
465543
.. rst-class:: sphx-glr-timing
466544

467-
**Total running time of the script:** ( 10 minutes 7.307 seconds)
545+
**Total running time of the script:** ( 9 minutes 57.077 seconds)
468546

469547

470548
.. _sphx_glr_download_advanced_tabular_example_resampling_strategy.py:

0 commit comments

Comments
 (0)