Skip to content

Improve mlos-viz for multiple repeats of a config and add tests #633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 137 commits into from
Jan 29, 2024
Merged

Improve mlos-viz for multiple repeats of a config and add tests #633

merged 137 commits into from
Jan 29, 2024

Conversation

bpkroth
Copy link
Contributor

@bpkroth bpkroth commented Jan 16, 2024 •
edited
Loading

  • Mark mlos_viz as typed for mypy
  • Bump version
  • Mock calls to matplotlib/dabl for testing
  • Add plotting of top-N configs
  • Improve plots for handling repeat config trials via variance error bars

@bpkroth bpkroth added WIP Work in progress - do not merge yet tests Add or fix unit tests labels Jan 16, 2024
@bpkroth bpkroth changed the title Add tests for mlos-viz Add tests for TrialData and mlos-viz Jan 16, 2024
@bpkroth bpkroth marked this pull request as ready for review January 26, 2024 21:11
@bpkroth bpkroth requested a review from a team as a code owner January 26, 2024 21:11
@bpkroth bpkroth added enhancement New feature or request ready for review Ready for review and removed WIP Work in progress - do not merge yet labels Jan 26, 2024
motus pushed a commit that referenced this pull request Jan 26, 2024
@bpkroth
Add default_tunable_config_id property to ExperimentData (#650)
56e84fd 
Splitting work out from #633.

Having an easy way to identify what is the "default" config group for an
Experiment makes analysis of improvements easier.

One challenge with this is that we don't currently strictly enforce
non-changes between resumptions of an Experiment, so it's currently
possible for the user to change the tunables config and adjust what the
"default".

Moreover, in the past we didn't store what the default config was in the
DB.

By convention we take the first one for now as a fallback.
motus added a commit that referenced this pull request Jan 26, 2024
@bpkroth @motus
Fixes to max_iterations calculation when `trial_config_repeat_count…
d073fc1 
…` is given. (#651)

Splitting work out from #633.
Bug fix for #642.

Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
motus
motus approved these changes Jan 29, 2024
View reviewed changes
Copy link
Member

@motus motus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@motus motus enabled auto-merge (squash) January 29, 2024 01:46
@motus motus merged commit a45f97d into microsoft:main Jan 29, 2024
@bpkroth bpkroth deleted the mlos_viz-testing branch February 27, 2024 23:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@motus motus motus approved these changes

Assignees

No one assigned

Labels

enhancement New feature or request mlos-bench mlos-viz ready for review Ready for review tests Add or fix unit tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@bpkroth @motus