Multi Objective CNN benchmark - [WIP] #147
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ayushi-3536
commented
May 6, 2022
ayushi-3536
commented
- Added benchmark outline for CNN benchmarks from paper
- Pull from MO interface by Phillip is pending
PhMueller
reviewed
May 6, 2022
PhMueller
reviewed
May 6, 2022
PhMueller
reviewed
May 6, 2022
PhMueller
reviewed
May 6, 2022
PhMueller
reviewed
May 6, 2022
PhMueller
reviewed
May 6, 2022
PhMueller
reviewed
May 6, 2022
- changed lock name for mo_cnn bench
- removed hard coded model input to support multiple datasets
This reverts commit f76ea99.
…ation of various datasets - changed epoch from 50 to 25 (from literature) - corrected epoch training(0 indexed) - removed subsample from fidelity(not done in literature, can discuss to add it if we want to perform experiments for this) - returning python object
PhMueller
reviewed
May 17, 2022
PhMueller
reviewed
May 17, 2022
PhMueller
reviewed
May 17, 2022
PhMueller
reviewed
May 17, 2022
PhMueller
reviewed
May 17, 2022
PhMueller
reviewed
May 17, 2022
PhMueller
reviewed
May 17, 2022
PhMueller
reviewed
May 17, 2022
PhMueller
reviewed
May 17, 2022
Codecov Report
@@ Coverage Diff @@
## development #147 +/- ##
===============================================
- Coverage 44.26% 42.23% -2.04%
===============================================
Files 41 46 +5
Lines 2415 2671 +256
===============================================
+ Hits 1069 1128 +59
- Misses 1346 1543 +197
|
- merged fidelity space and choice method
|
@KEggensperger, could you please have a look at it? |
KEggensperger
approved these changes
May 27, 2022
| val_accuracy = model.eval_fn(ds_val, device).item() | ||
| eval_valid_runtime = time.time() - start | ||
| start = time.time() | ||
| test_accuracy = model.eval_fn(ds_test, device).item() |
There was a problem hiding this comment.
Same questions as for the other benchmark: Why spending time on computing test metrics?
There was a problem hiding this comment.
Good question. Changed it to "training time".
The eval time should be almost equal for every run, so i think it is more important to report the "training time" instead of the "total time per configuration".
Thanks for the feedback!
-update test case
PhMueller
added a commit
to PhMueller/HPOBench
that referenced
this pull request
Feb 21, 2023
Added mo cnn benchmarks from bag of baseline paper We deviate from the original benchmark in two points: * we return as cost only the training time instead of the total elapsed time * we return as objective for minimization instead of `-100 * accuracy` now `1 - accuracy` to achieve better output scalings. Co-authored-by: ayushi-3536 <ayushi-3536@github.com> Co-authored-by: Philipp Müller <muller-phil@gmx.net>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.