[Core] Add docker run options

[Michaelvll]

This PR adds a new field allowing users to customize the run_options for the docker image.

docker:
  run_options:
      - -v /var/run/docker.sock:/var/run/docker.sock

TODO:
- [ ] propagate the run_options for k8s. (We cannot apply docker run_options to k8s pod containers)

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
  • sky launch -c test-dind --cloud aws --cpus 2 --image-id docker:continuumio/miniconda3:24.1.2-0 "apt update; apt install -y docker.io; docker run hello-world" with config above.
  • sky launch -c test-dind --cloud gcp --cpus 2 --image-id docker:continuumio/miniconda3:24.1.2-0 "apt update; apt install -y docker.io; docker run hello-world" with config above.
  • sky launch -c test-dind --cloud azure --cpus 2 --image-id docker:continuumio/miniconda3:24.1.2-0 "apt update; apt install -y docker.io; docker run hello-world" with config above.
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

[romilbhardwaj]

Thanks @Michaelvll! - haven't tried but the code lgtm. Left some comments on the documentation.

[romilbhardwaj]

Thanks @Michaelvll! LGTM - this will be quite helpful for our users.

Nit - bad args in run_config aren't surfaced nicely, but can be debugged by the user if they go through the logs. We may want to improve the error surfacing here in the future. Example.

[cblmemo]

Thanks for adding this @Michaelvll ! It would be useful for our user :) Left one thing to discuss and other parts looks great to me!

[cblmemo]

Where have we guarantee this "cannot be override"? In the code it seems like we are just appending them in the docker run cmds and follow the default behaviour of multiple same argument to the command. Do we need some checks on the docker run options to make sure it does not conflict w/ existing options?

[Michaelvll]

It is hard to check the arguments for docker. It might be fine for now to allow the program to proceed and let it error out after the cluster is launched. Since we have allowed the change of run_options for an existing cluster, a user can quickly fix it according to the error message. Wdyt?

[cblmemo]

sounds good!

[Michaelvll]

Thanks @Michaelvll! LGTM - this will be quite helpful for our users.

Nit - bad args in run_config aren't surfaced nicely, but can be debugged by the user if they go through the logs. We may want to improve the error surfacing here in the future. Example.

The error surfacing should be better now, by avoiding unnecessary retries and printing out the error:

$ sky launch -c doc --cloud aws --image-id docker:continuumio/miniconda3 "apt update; apt install -y docker.io; docker run hello-world"
I 06-26 08:30:01 provisioner.py:458] Successfully provisioned or found existing instance.
E 06-26 08:31:00 subprocess_utils.py:84] NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
E 06-26 08:31:00 subprocess_utils.py:84] 
E 06-26 08:31:00 subprocess_utils.py:84] 
E 06-26 08:31:00 subprocess_utils.py:84] unknown flag: --zhwu
E 06-26 08:31:00 subprocess_utils.py:84] See 'docker run --help'.
E 06-26 08:31:00 subprocess_utils.py:84] 
E 06-26 08:31:00 provisioner.py:591] *** Failed setting up cluster. ***
Clusters
NAME  LAUNCHED    RESOURCES                                                                  STATUS  AUTOSTOP  COMMAND                           
doc   2 mins ago  1x AWS(m6i.2xlarge, image_id={'us-east-1': 'docker:continuumio/minicon...  INIT    -         sky launch -c doc --cloud aws...  

sky.exceptions.CommandError: Command docker run --name sky_container -d -it -e LC_ALL=C.UTF-8 -e LANG=C.UTF-8 --ulimit nofile=1048576:1048576 -v /var/run/docker.sock:/var/run/docker.sock --zhwu=True --shm-size=2g --net=host --cap-add=SYS_ADMIN --device=/dev/fuse --security-opt=apparmor:unconfined continuumio/miniconda3 bash failed with return code 125.
Failed to run docker setup commands.

Remove the problematic arguments, and sky launch again on the existing cluster, and it works.

Tested:

• sky launch -c doc --cloud aws --image-id docker:continuumio/miniconda3 "apt update; apt install -y docker.io; docker run hello-world" with

  docker:
    run_options:
        - -v /var/run/docker.sock:/var/run/docker.sock
        - --shm-size=2g
  

• sky launch -c doc --cloud aws --image-id docker:continuumio/miniconda3 "echo hi" without config.yaml