[Fluidstack] Improve fluidstack provisioner for corner cases #3254
Conversation
sky/provision/fluidstack/instance.py
Outdated
| head_instance_id = inst_id | ||
| rename(inst_id, f'{cluster_name_on_cloud}-head') | ||
| logger.info(f'Rename head node {head_instance_id}.') | ||
| if inst_id != head_instance_id and inst['hostname'].endswith('-head'): |
There was a problem hiding this comment.
Is this for the case when launching on an existing cluster where some node has been set to head node? in this case, should we keep the original head node?
There was a problem hiding this comment.
This is for the case when there are multiple instances with -head. We should have reuse the same head instance if only one head exist in L102
There was a problem hiding this comment.
Why would there be multiple -head? A bug on the CSP side? Worth a comment?
There was a problem hiding this comment.
Just added a comment about this. This should not happen in most of the case. It is just for a safeguard against the case when somehow multiple head instances exist, e.g., a user changed the name manually.
sky/provision/fluidstack/instance.py
Outdated
| head_instance_id = inst_id | ||
| rename(inst_id, f'{cluster_name_on_cloud}-head') | ||
| logger.info(f'Rename head node {head_instance_id}.') | ||
| if inst_id != head_instance_id and inst['hostname'].endswith('-head'): |
There was a problem hiding this comment.
Why would there be multiple -head? A bug on the CSP side? Worth a comment?
…o improve-fluidstack
|
Just updated the instance creation logic, so that we don't wait infinitely for a instance directly being converted to terminated state from a pending state. PTAL @concretevitamin @cblmemo |
|
Merging this now to unblock our user : ) |
…o improve-fluidstack
Fixes #3259.
It also fixes the issue when there are multiple existing instance with the name
*-head, we now reuse those instances by renaming them.Tested (run the relevant ones):
bash format.shsky launch --cloud fluidstack --gpus A100-80GB nvidia-smi -c test-fluidsky launch -c test-a100 --cloud fluidstack --gpus A100-80GB nvidia-smi --num-nodes 2sky launch -c test-a100 --cloud fluidstack --gpus A100-80GB:8 nvidia-smisky launch -c test-h100 --cloud fluidstack --gpus H100 nvidia-smisky launch -c test-h100 --cloud fluidstack --gpus H100:2 nvidia-smisky launch -c test-h100 --cloud fluidstack --gpus H100:8 nvidia-smipytest tests/test_smoke.pypytest tests/test_smoke.py::test_fill_in_the_namebash tests/backward_comaptibility_tests.sh