[Core] Command runner refactor and avoid source ~/.bashrc for better speed

[Michaelvll]

Separated from #3157.

1. This refactors the command runner for the future support of other command runners, including kubernetes and slurm
2. This PR avoids source bashrc for internal remote command execution to speed up job submission and launching.

Compared to master (8a18133)

Launching new clusters

GCP (new provisioner)

for i in `seq 1 5`; do sky launch -y -c test-dep-$i --cloud gcp --cpus 2 echo hi; done

Master: (145+130+163+127+145)/5 = 142s = 2m22s
This PR: (134+123+132+134+142)/5 = 133 = 2m13s (-9s)

Azure (ray autoscaler)

for i in `seq 1 5`; do sky launch -y -c test-azure-$i --cloud azure --cpus 2 echo hi; done

Master: (514 + 514+514+521+521)/5 = 517s = 8m37s
This PR: (514 + 511 + 512+510+544)/5 = 518s = 8m38s

Kubernetes (compared to 12c156a)

for i in `seq 1 5`; do sky launch -y -c test-dep-$i --cloud kubernetes --cpus 2 echo hi; done

Master: (66 + 65 + 65 + 60 + 62) / 5 = 63.6s
This PR: (60 + 59 + 62 + 58 + 58) / 5 = 59.4s (-4.2s)

Exec on existing cluster

Kubernetes (compared to 12c156a)

for i in `seq 1 5`; do time sky exec test-exec echo hi -d; done

Master: (11.89 + 12.36 + 11.77 + 12.42 + 12.27) / 5 = 12.14s
This PR: (9.87 + 9.44 + 9.07 + 9.78 + 9.49) / 5 = 9.53s (-2.61s)

Launching new cluster with file mounts

resources:
  cpus: 1
  # image_id: docker:michaelvll/no-local-bin-test:v3
  # image_id: docker:michaelvll/k8s-dep-test:v2

file_mounts:
  /sky/examples: ./examples
  ~/.viminfo : ~/.viminfo
  ~/task.yaml : ./task.yaml
  ~/.config/gcloud: ~/.config/gcloud

run: |
  echo hi

for i in `seq 1 5`; do sky launch -y -c test-dep-$i --cloud gcp --cpus 2 task.yaml; done

Master: (147 + 124 + 149 + 150 + 147) / 5 = 143.5s = 2m23.5s
This PR: (145 + 140 + 139 + 142+153)/5 = 143.8s = 2m23.8s

Launching on an existing cluster

for i in `seq 1 5`; do sky launch -y -c test-exist echo hi; done

Master: (41+40+40+40+51)/5 = 42.4s
This PR: (39+38+37+37+38)/5 = 37.8s (-4.6s)

Exec on an existing cluster

for i in `seq 1 5`; do sky exec test-exist echo hi; done

Master: (8+6+7+7+7)/5 = 7s
This PR: (6+7+6+6+6)/5 = 6.2s (-0.8s)

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

[cblmemo]

Thanks! It looks mostly good to me. Left several nits and it should ready to go after smoke test 🫡

[Michaelvll]

Tested (65001c8):

• pytest tests/test_smoke.py
• pytest tests/test_smoke.py --aws
• pytest tests/test_smoke.py --kubernetes
• bash -i ./tests/backward_compatibility_tests.sh