[k8s] Share SSH jump pod #2826

kbrgl · 2023-11-28T04:28:41Z

This PR closes #2598 and adds plumbing to share the SSH jump pod across a Kubernetes namespace.

Per-user secrets for the user's public SSH key are replaced with a single secret object, sky-ssh-key, that is created once and patched when future attempts to setup Kubernetes authentication are made.
For initial jump pod creation, the manifest runs cat /etc/secret-volume/ssh-key-* > ~/.ssh/authorized_keys to set up the initial user's SSH key.
When a new user tries to authenticate, the patched secret is re-mounted at /etc/secret-volume and this change is picked up by the LCM script which reruns cat /etc/secret-volume/ssh-key-* > ~/.ssh/authorized_keys.

The LCM script now uses 2 threads to poll: one to manage the lifecycle and terminate the jump pod, and one to reload keys on an interval.

Testing

Tested (run the relevant ones):

Code formatting: bash format.sh
Any manual or new tests for this PR (please specify below)
All smoke tests: pytest tests/test_smoke.py
Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
Backward compatibility tests: bash tests/backward_compatibility_tests.sh

Tested locally.

cc @romilbhardwaj

was causing concatentation issues

romilbhardwaj

Awesome, thanks @kbrgl! Took a skim through the code and it looks good. Left some nits. Will test it and approve if all is good.

sky/utils/kubernetes_utils.py

sky/clouds/kubernetes.py

sky/utils/kubernetes/ssh_jump_lifecycle_manager.py

Co-authored-by: Romil Bhardwaj <romil.bhardwaj@gmail.com>

romilbhardwaj

Thanks @kbrgl! Did some manual testing - seems to be working nicely. Backward compatibility has some issues, see comments.

sky/utils/kubernetes/ssh_jump_lifecycle_manager.py

sky/authentication.py

sky/utils/kubernetes/ssh_jump_lifecycle_manager.py

Co-authored-by: Romil Bhardwaj <romil.bhardwaj@gmail.com>

…ilot into kabir/issue-2598-jump # Conflicts: # sky/clouds/kubernetes.py # sky/templates/kubernetes-ray.yml.j2

…o kabir/issue-2598-jump # Conflicts: # sky/provision/kubernetes/instance.py # sky/provision/kubernetes/utils.py

romilbhardwaj

Thanks @kbrgl! Pushed some nits and merge conflict fixes. Manually tested backward compat, now running smoke tests.

Once everything is ready, will merge this PR and push new container images to our image registry.

…o kabir/issue-2598-jump

romilbhardwaj

Thanks @kbrgl! Smoke tests pass. Manually tested backward compatibility. Building and updating our docker image, will merge after that's done.

…the base ray image

romilbhardwaj · 2024-02-21T20:36:31Z

Updated images and build script to use date versioning - [K8s] Use timestamped k8s image versions rather than latest skypilot-catalog#60
Fixed [k8s] boto3 from base GPU image is stale, causes test_huggingface to fail #3158
Added socat and netcat to base image on user request for faster provisioning

Running k8s smoke tests - smoke tests pass, merging now.

kbrgl added 7 commits November 23, 2023 15:05

[k8s] use constant jump pod name for namespacing

ee8f113

wip: use single secret object to store jump keys

56f8e99

wip: reload SSH keys on interval in jump lifecycle manager

9afb1ca

fix import

f3004e7

fix: ensure public key ends with newline

eb870c6

was causing concatentation issues

fix comment

c05ce9e

use threading to allow differing poll intervals in LCM script

4e2b661

kbrgl changed the title ~~Share k8s SSH jump pod~~ [k8s] Share k8s SSH jump pod Dec 20, 2023

kbrgl marked this pull request as ready for review December 20, 2023 09:09

kbrgl changed the title ~~[k8s] Share k8s SSH jump pod~~ [k8s] Share SSH jump pod Dec 20, 2023

romilbhardwaj reviewed Dec 23, 2023

View reviewed changes

kbrgl and others added 4 commits December 24, 2023 10:58

Use timeout in first kubeapi call

108ff89

Co-authored-by: Romil Bhardwaj <romil.bhardwaj@gmail.com>

Use sky-ssh-keys instead of sky-ssh-key

5053296

Co-authored-by: Romil Bhardwaj <romil.bhardwaj@gmail.com>

Update LCM manager docstring

4e502a9

Co-authored-by: Romil Bhardwaj <romil.bhardwaj@gmail.com>

Clean up error messages in LCM

1f85115

Co-authored-by: Romil Bhardwaj <romil.bhardwaj@gmail.com>

kbrgl requested a review from romilbhardwaj December 24, 2023 05:33

romilbhardwaj reviewed Dec 26, 2023

View reviewed changes

sky/utils/kubernetes/ssh_jump_lifecycle_manager.py Outdated Show resolved Hide resolved

sky/authentication.py Outdated Show resolved Hide resolved

sky/utils/kubernetes/ssh_jump_lifecycle_manager.py Outdated Show resolved Hide resolved

romilbhardwaj mentioned this pull request Dec 26, 2023

[k8s] Backward compatibility between SkyPilot version and k8s image #2901

Closed

kbrgl added 6 commits January 5, 2024 18:45

add parent=skypilot label to secret object

b96bc50

diff keys and only reload if changed

4f18307

more accurate error message

190994a

backwards compatibility fix

8eff1c6

Run yapf and pylint

804bf98

fix format

ae7683d

kbrgl requested a review from romilbhardwaj January 6, 2024 12:32

Michaelvll force-pushed the master branch from 71213e5 to 9743aa0 Compare January 13, 2024 05:30

Merge branch 'master' into kabir/issue-2598-jump

1be67fc

romilbhardwaj mentioned this pull request Feb 7, 2024

[k8s] Prototype - SkyServe on K8s #3109

Closed

14 tasks

kbrgl added 2 commits February 6, 2024 18:14

[k8s] use constant jump pod name for namespacing

ae4a3ac

wip: use single secret object to store jump keys

c80173f

kbrgl and others added 8 commits February 7, 2024 12:38

Update LCM manager docstring

e169fb4

Co-authored-by: Romil Bhardwaj <romil.bhardwaj@gmail.com>

Clean up error messages in LCM

f297ac4

Co-authored-by: Romil Bhardwaj <romil.bhardwaj@gmail.com>

add parent=skypilot label to secret object

850f7d2

diff keys and only reload if changed

0290a6b

more accurate error message

fa1bcc1

backwards compatibility fix

8e3e8e7

Run yapf and pylint

e0474e8

fix format

e125c05

kbrgl force-pushed the kabir/issue-2598-jump branch from 1be67fc to e125c05 Compare February 7, 2024 20:43

kbrgl added 2 commits February 7, 2024 12:50

fix rebase mistakes

afcdd84

fix ssh provision issue by reordering perm changes

77258da

romilbhardwaj added this to the v0.5 milestone Feb 8, 2024

run lint and format

0acf179

romilbhardwaj mentioned this pull request Feb 14, 2024

[k8s] boto3 from base GPU image is stale, causes test_huggingface to fail #3158

Closed

romilbhardwaj added 6 commits February 20, 2024 15:01

Merge branch 'kabir/issue-2598-jump' of https://github.com/kbrgl/skyp…

8d0e94b

…ilot into kabir/issue-2598-jump # Conflicts: # sky/clouds/kubernetes.py # sky/templates/kubernetes-ray.yml.j2

Merge branch 'master' of https://github.com/skypilot-org/skypilot int…

59e4c96

…o kabir/issue-2598-jump # Conflicts: # sky/provision/kubernetes/instance.py # sky/provision/kubernetes/utils.py

move auth key cat up

4bc30f2

move auth key cat down

76e62fd

remove logger

ffb057e

lint

13aed4f

romilbhardwaj reviewed Feb 21, 2024

View reviewed changes

romilbhardwaj added 3 commits February 20, 2024 17:14

Merge branch 'master' of https://github.com/skypilot-org/skypilot int…

1837937

…o kabir/issue-2598-jump

Add missed commit from skypilot-org#200

74e51da

Merge branch 'master' of https://github.com/skypilot-org/skypilot int…

2a65bac

…o kabir/issue-2598-jump

romilbhardwaj approved these changes Feb 21, 2024

View reviewed changes

romilbhardwaj added 2 commits February 21, 2024 10:39

update build_image to use latest or date tag

25b5db4

update docker images to install socat and netcat and update boto3 on …

a0dea6c

…the base ray image

romilbhardwaj mentioned this pull request Feb 21, 2024

[K8s] Use timestamped k8s image versions rather than latest skypilot-org/skypilot-catalog#60

Merged

4 tasks

romilbhardwaj merged commit f3f6566 into skypilot-org:master Feb 21, 2024
19 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[k8s] Share SSH jump pod #2826

[k8s] Share SSH jump pod #2826

kbrgl commented Nov 28, 2023 •

edited

Loading

romilbhardwaj left a comment

romilbhardwaj left a comment

romilbhardwaj left a comment

romilbhardwaj left a comment

romilbhardwaj commented Feb 21, 2024 •

edited

Loading

[k8s] Share SSH jump pod #2826

[k8s] Share SSH jump pod #2826

Conversation

kbrgl commented Nov 28, 2023 • edited Loading

Testing

romilbhardwaj left a comment

Choose a reason for hiding this comment

romilbhardwaj left a comment

Choose a reason for hiding this comment

romilbhardwaj left a comment

Choose a reason for hiding this comment

romilbhardwaj left a comment

Choose a reason for hiding this comment

romilbhardwaj commented Feb 21, 2024 • edited Loading

kbrgl commented Nov 28, 2023 •

edited

Loading

romilbhardwaj commented Feb 21, 2024 •

edited

Loading