[k8s] sky status --k8s refactor
#4079
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Michaelvll
reviewed
Oct 13, 2024
There was a problem hiding this comment.
Thanks for the refactoring @romilbhardwaj! Left a few comments : )
sky/core.py
Outdated
| 'launched_at': Timestamp of when the cluster was launched | ||
| 'resources': sky.Resources object for the cluster | ||
| """ | ||
| return backend_utils.get_clusters_kubernetes() |
There was a problem hiding this comment.
nit: It seems the get_clusters_kubernetes() is only called here. It might be better to keep it here instead of having a shallow call to the backend_utils, especially backend_utils contains too many things right now.
There was a problem hiding this comment.
Agreed, I was previously following the same code path pattern as sky.status(), but it makes sense to reduce shallow functions. Updated.
sky/core.py
Outdated
Comment on lines
123
to
125
| - List of dictionaries with cluster information. | ||
| - Dictionary of job controller information. | ||
| - Dictionary of serve controller information. |
There was a problem hiding this comment.
Why do we use List for the first return, but Dict for the following? Can we keep them consistent, e.g., using List for all of them?
Also, using dict for the information is a bit fragile. I would vote to gradually change everything to dataclasses.dataclass or even pydantic.BaseModel. Could we use dataclass for the returned value of this (and the underlying function)?
There was a problem hiding this comment.
Good point - updated to return List in all cases. Also updated to use dataclass, was previously trying to be consistent with the return type from backend_utils.get_clusters :)
Michaelvll
approved these changes
Oct 14, 2024
Michaelvll
reviewed
Oct 14, 2024
Michaelvll
reviewed
Oct 15, 2024
sky/cli.py
Outdated
Comment on lines
1472
to
1474
| for i, job_controller_info in enumerate(jobs_controllers): | ||
| user = job_controller_info.user | ||
| pod = job_controller_info.pods[0] |
There was a problem hiding this comment.
For the spinner below, please use ux_utils.spinner_message() for the style of the messaage
|
Thanks @Michaelvll! Comments should be resolved now, running a few final manual tests before merging. |
|
Merging now! |
cblmemo
added a commit
that referenced
this pull request
Oct 21, 2024
…t Flow Definition (#4067) * provide an example, edited from pipeline.yml * more focus on dependencies for user dag lib * more powerful user interface * load and dump new yaml format * fix * fix: reversed logic in add_edge * [docs] Unroll k8s internal load balancer docs (#4083) unroll load balancer docs * rename * refactor due to reviewer's comments * generate task.name if not given * [docs] `sky status --kubernetes` docs (#4064) * observability docs * comments * [UX] Show log after failure and fix the color issue with narrow window (#4084) * fix narrow window and show log path during exception * format * format * [k8s] `sky status --k8s` refactor (#4079) * refactor * lint * refactor, dataclass * refactor, dataclass * refactor * lint * add comments for add_edge * add `print_exception_no_traceback` when raise * make `Dag.tasks` a property * print dependencies for `__repr__` * move `get_unique_task_name` to common_utils * [Performance] Use new GCP custom images (#4027) * [Performance] Use new custom image to create GCP GPU VMs * update image tags for both CPU and GPU * always generate .sky/python_path --------- Co-authored-by: Yika Luo <yikaluo@Yikas-MacBook-Pro.local> * [GCP] Add H100 mega (#4099) * Add H100 mega support on GCP * fix for some other regions * format * fix resource type * fix catalog fetching * [GCP] Add gVNIC support (#4095) * add gvnic support through config.yaml * lint * docs * [Lambda] Lambda Cloud SkyPilot provisioner (#3865) * feat: lambda cloud new provisioner * feat: address cblmemo reviews and other reviews + make multi-node work again * fix: quotes * fix: address some reviews * chore: rm unused option * chore: update typedef * feat: use lists directly * fix: formatting * chore: address reviews * fix: formatting * chore: rm query ports since default impl per review * feat: add back query ports * fix: formatting * chore: add newline at eof * feat: try removing query ports again * [Docs] GKE Nvidia Driver installation instructions update (#4106) * docs * docs * docs * [Performance] Use new AWS custom images (#4091) * rename methods to use downstream/edge terminology * [Performance] Add Packer image generation scripts for GCP and AWS (#4068) * [Performance] Add Packer image generation scripts for GCP and AWS * Add docker install and tests * solve nvidia container issue * Install cuDNN * [Performance] Scripts to copy/delete AWS images for all regions and add cloud deps (#4073) * [Performance] Add AWS script to copy images for all regions * script to delete all AWS images across regions * Add cloud dependencies to image --------- Co-authored-by: Yika Luo <yikaluo@Yikas-MacBook-Pro.local> * Disable AWS images.csv refreshing (#4116) * [Docs] .skyignore doc (#4114) * [Docs] .skyignore doc * Correct typos Co-authored-by: Zongheng Yang <zongheng.y@gmail.com> --------- Co-authored-by: Zongheng Yang <zongheng.y@gmail.com> * [Core] Raise error for none existing cluster when endpoint is called (#4117) raise error for none existing cluster * Refresh local aws images.csv when image not found (#4127) Refresh local aws images.csv by pulling from github catalog when image tag not found * [Docs] News revamps. (#4126) * News revamps. updates updates updates updates updates updates updates updates * Apply suggestions from code review Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com> --------- Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com> * [Serve] Support manually terminating a replica and with purge option (#4032) * define replica id param in cli * create endpoint on controller * call controller endpoint to scale down replica * add classmethod decorator * add handler methods for readability in cli * update docstr and error msg, and inline in cli * update log and return err msg * add docstr, catch and reraise err, add stopped and nonexistent message * inline constant to avoid circular import * fix error statement and return encoded str * add purge feature * add purge replica usage in docstr * use .get to handle unexpected packages * fix: diff terminate replica when failed/purging or not * fix: stay up to date for `is_controller_accessible` * revert * up to date with current APIs * error handling * when purged remove record in the main loop * refactor due to reviewer's suggestions * combine functions * fix: terminate the healthy replica even with purge option * remove abbr * Update sky/serve/core.py Co-authored-by: Tian Xia <cblmemo@gmail.com> * Update sky/serve/core.py Co-authored-by: Tian Xia <cblmemo@gmail.com> * Update sky/serve/controller.py Co-authored-by: Tian Xia <cblmemo@gmail.com> * Update sky/serve/controller.py Co-authored-by: Tian Xia <cblmemo@gmail.com> * Update sky/cli.py Co-authored-by: Tian Xia <cblmemo@gmail.com> * got services hint * check if not yes in the outside if branch * fix some output messages * Update sky/serve/core.py Co-authored-by: Tian Xia <cblmemo@gmail.com> * set conflict status code for already scheduled termination * combine purge and normal terminating down branch together * bump version * global exception handler to render a json response with error messages * fix: use responses.JSONResponse for dict serialize * error messages for old controller * fix: check version mismatch in generated code * revert mistakenly change update_service * refine already in terminating message * fix: branch code workaround in cls.build * wording Co-authored-by: Tian Xia <cblmemo@gmail.com> * refactor due to reviewer's comments * fix use ux_utils Co-authored-by: Tian Xia <cblmemo@gmail.com> * add changelog as comments * fix messages * edit the message for mismatch error Co-authored-by: Tian Xia <cblmemo@gmail.com> * no traceback when raising in `terminate_replica` * messages decode * Apply suggestions from code review Co-authored-by: Tian Xia <cblmemo@gmail.com> * format * forma * Empty commit --------- Co-authored-by: David Tran <davidtran@Davids-MacBook-Pro-2.local> Co-authored-by: David Tran <david.tran@datadoghq.com> Co-authored-by: Tian Xia <cblmemo@gmail.com> * [Provisioner] Support docker in Lambda Cloud and TPU (#4115) * [Provisioner] Support docker in Lambda Cloud * fix permission issue * merge with check docker installed * add tpu support & test * patch lambda cloud * add comment * Apply suggestions from code review Co-authored-by: Tian Xia <cblmemo@gmail.com> * change wording all to up/downstream style * Add unique suffix to task names, fallback to timestamp if unnamed * Unify handling of single and multiple tasks without dependencies * Refactor tasks initialization: use list comprehension and fail fast * Fix remove task dependency description: upstream, not downstream Co-authored-by: Tian Xia <cblmemo@gmail.com> * Remove duplicated `self.edges`, use nx api instead * [Serve] Add `ux_utils.print_exception_no_traceback()` for cleaner error output (#4111) * add `ux_utils.print_exception_no_traceback()` for cleaner error output * Empty commit * remove unnecessary with block * Partially revert: Remove unnecessary `ux_utils.print_exception_no_traceback()` wrappers (#4130) fix unnecessary with block for returning * Revert "Add unique suffix to task names, fallback to timestamp if unnamed" Otherwise, users can not refer to the task by name in the DAG. This reverts commit 8486352. * comment the checking used as upstream logic * [examples] Deepspeed fixes + k8s support (#4124) deepspeed kubernetes fixes * Empty commit * [OCI] Support more OS types in addition to ubuntu (#4080) * Bug fix for sky config file path resolution. * format * [OCI] Bug fix for image_id in Task YAML * [OCI]: Support more OS types (esp. oraclelinux) in addition to ubuntu. * format * Disable system firewall * Bug fix for validation of the Marketplace images * Update sky/clouds/oci.py Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com> * Update sky/clouds/oci.py Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com> * variable/function naming * address review comments: not to change the service_catalog api. call oci_catalog directly for get os type for a image. * Update sky/clouds/oci.py Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com> * Update sky/clouds/oci.py Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com> * Update sky/clouds/oci.py Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com> * address review comments --------- Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com> * Apply suggestions from code review Co-authored-by: Tian Xia <cblmemo@gmail.com> * fix: typing.cast * add TODOs for future function migration * remove dependencies wording to reduce ambiguity * temporarily add github actions --------- Co-authored-by: Romil Bhardwaj <romil.bhardwaj@berkeley.edu> Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com> Co-authored-by: yika-luo <yikaluo@assemblesys.com> Co-authored-by: Yika Luo <yikaluo@Yikas-MacBook-Pro.local> Co-authored-by: Kote Mushegiani <mushegiani@gmail.com> Co-authored-by: Zongheng Yang <zongheng.y@gmail.com> Co-authored-by: David Tran <davidtran@Davids-MacBook-Pro-2.local> Co-authored-by: David Tran <david.tran@datadoghq.com> Co-authored-by: Tian Xia <cblmemo@gmail.com> Co-authored-by: Hysun He <hysunhe@foxmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Follow up to #4040. Refactors cli.py and status_utils.py to remove dependency on kubernetes_utils for
sky status --k8s.