Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Disable spot instances for training #376

Merged
merged 2 commits into from
Jan 26, 2024
Merged

Disable spot instances for training #376

merged 2 commits into from
Jan 26, 2024

Conversation

gabrielBusta
Copy link
Member

fixes #356

@gabrielBusta gabrielBusta requested a review from bhearsum January 19, 2024 20:58
@gabrielBusta gabrielBusta requested a review from a team as a code owner January 19, 2024 20:58
bhearsum
bhearsum approved these changes Jan 19, 2024
View reviewed changes
Copy link
Collaborator

@bhearsum bhearsum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! I suggest that you double check that these instances come up correctly on a branch before merging.

@gabrielBusta
Copy link
Member Author

@bhearsum I think there might be an issue. I haven't seen the new instances pick up the tasks in the PR and its been about 40 minutes

@gabrielBusta
Copy link
Member Author

https://firefox-ci-tc.services.mozilla.com/tasks/Sq2XaUq6RxWY8be0VA03jQ

@bhearsum
Copy link
Collaborator

https://firefox-ci-tc.services.mozilla.com/tasks/Sq2XaUq6RxWY8be0VA03jQ

I ended up canceling the pending training task in this task group due to https://bugzilla.mozilla.org/show_bug.cgi?id=1875371#c9

@bhearsum
Copy link
Collaborator

Also a reminder: we need to put together a list of what we need to turn spot instances back on, perhaps as a new issue in this repo?

@gabrielBusta gabrielBusta force-pushed the use-standard-vms-for-training branch from 32dc99b to cd0d6d6 Compare January 25, 2024 20:45
@gabrielBusta
Copy link
Member Author

Looks like the non-spot instances are picking up tasks now!

https://firefox-ci-tc.services.mozilla.com/tasks/GpiEGvO0TcadEyvC_kn5Fg

@gabrielBusta gabrielBusta force-pushed the use-standard-vms-for-training branch from cd0d6d6 to 6c9bbe7 Compare January 25, 2024 21:42
@gabrielBusta gabrielBusta force-pushed the use-standard-vms-for-training branch from 6c9bbe7 to eecb51a Compare January 26, 2024 20:32
@gabrielBusta gabrielBusta merged commit 26752d0 into main Jan 26, 2024
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bhearsum bhearsum bhearsum approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Disable spot instances for training
2 participants
@gabrielBusta @bhearsum