[Core] Allow disabling ECC for nvidia-gpu #3676
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sky/backends/backend_utils.py
Outdated
Comment on lines
879
to
880
| if skypilot_config.get_nested(('nvidia_gpus', 'disable_ecc'), False): | ||
| initial_setup_commands.append(constants.DISABLE_GPU_ECC_COMMAND) |
There was a problem hiding this comment.
Should we append these commands only if disable_ecc is set to true AND to_provision contains GPU requests? Otherwise this will apply to CPU-only clusters too?
There was a problem hiding this comment.
Good point! Added an additional condidtion for checking the accelerators, though it should be fine even with that additional check for the CPU instances, as it will just skip the process without nvidia-smi available.
romilbhardwaj
approved these changes
Jun 20, 2024
Michaelvll
added a commit
that referenced
this pull request
Aug 23, 2024
* Disable ECC for nvidia-gpu * Add config.rst * format * address * Note for the reboot overhead * address comments * fix fluidstack * Avoid disable ecc for clouds using ray autoscaler due to the lack of retry after reboot
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
According to the following link, disabling ECC can improve the GPU performance by 30%, and a user has requested this knob. This is experimental and might be blocked by the task-level config, as the user would like to specify this per-task.
https://portal.nutanix.com/page/documents/kbs/details?targetId=kA00e000000LKjOCAW
Added config:
We only allow disabling ecc for the clouds with SkyPilot provisioner, as those with ray autoscaler does not handle the retry correctly.
Tested (run the relevant ones):
bash format.shsky launch --cloud aws --gpus t4 "nvidia-smi -q | grep 'ECC Mode' -A 2"with config set to turn off ecc.sky launch --cloud gcp --gpus t4 "nvidia-smi -q | grep 'ECC Mode' -A 2"with config set to turn off ecc.sky launch --cloud kubernetes --gpus t4 "nvidia-smi -q | grep 'ECC Mode' -A 2"with config set to turn off ecc. This will still launch the cluster though skipping the ECC disabling.sky launch --cloud runpod --gpus rtxa6000 "nvidia-smi -q | grep 'ECC Mode' -A 2"RunPod has ecc turned off by default.sky launch -c test-lam --down --cloud lambda --gpus A10 "nvidia-smi -q | grep 'ECC Mode' -A 2"skip the disabling due to the use of ray autoscalersky launch --cloud gcp --gpus t4 "nvidia-smi -q | grep 'ECC Mode' -A 2"without config set to turn off ecc.pytest tests/test_smoke.pypytest tests/test_smoke.py::test_fill_in_the_nameconda deactivate; bash -i tests/backward_compatibility_tests.sh