Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] Faster optimizer table by disabling reservation check #3280

Merged

Conversation

Michaelvll
Copy link
Collaborator

@Michaelvll Michaelvll commented Mar 5, 2024

Partially fixes #3159

Upper current PR, lower master (1.8s -> 0.5s): the time it takes for starting _launch_with_confirm to confirmation prompt.
image

On my local laptop, it takes even longer: 127s vs 2s
Upper master, lower currnet PR
image

The reservation check on GCP causes the optimizer table to show up very slowly. We now disable the check by default, and add a new option to turn on it.

This issue becomes significant when H100 is involved (#3279, need to confirm)

Tested (run the relevant ones):

  • Code formatting: bash format.sh
  • Any manual or new tests for this PR (please specify below)
  • All smoke tests: pytest tests/test_smoke.py
  • Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
  • Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

…-optimizer-table-by-disabling-reservation-check
Copy link
Member

@concretevitamin concretevitamin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, seeing >2x speedup on my end too.

Master

sky launch:
5.59, 3.84, 5.64

sky launch --gpus H100:8
3.19, 3.03, 3.17

#3280

sky launch:
2.38, 2.62, 2.21

sky launch --gpus H100:8
1.75, 1.41, 1.44

docs/source/reference/config.rst Show resolved Hide resolved
docs/source/reference/config.rst Outdated Show resolved Hide resolved
docs/source/reference/config.rst Outdated Show resolved Hide resolved
Michaelvll and others added 3 commits March 6, 2024 22:39
Co-authored-by: Zongheng Yang <zongheng.y@gmail.com>
Co-authored-by: Zongheng Yang <zongheng.y@gmail.com>
docs/source/reference/config.rst Outdated Show resolved Hide resolved
@Michaelvll Michaelvll merged commit a7fdfd2 into master Mar 7, 2024
19 checks passed
@Michaelvll Michaelvll deleted the faster-optimizer-table-by-disabling-reservation-check branch March 7, 2024 23:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

sky launch takes ~5s to print out optimizer table, which is slow
2 participants