[UX] Better logging message for operators on the clusters terminated manually in the cloud

[Michaelvll]

Fixes #2298.

Note this only works for the spot clusters or the clusters with autostop/down setup, as the status refresh will not be triggered for the operators on the normal clusters.

Previously:

sky logs test-warning-spot
ValueError: Cluster 'test-warning-spot' does not exist.

Now:

sky logs test-warning-spot
ValueError: Cluster 'test-warning-spot' was preempted or manually terminated in console.

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
  • sky launch -c test-warning-spot --cpus 2 --cloud gcp --use-spot
  • manually terminate the cluster in the console
  • sky logs test-warning-spot
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

[concretevitamin]

Nice @Michaelvll! QQ: does this cover all ops that auto refresh, e.g., queue as mentioned in the issue?

[Michaelvll]

Nice @Michaelvll! QQ: does this cover all ops that auto refresh, e.g., queue as mentioned in the issue?

Yes, it works for all the operators with the auto refresh.

One thing to note is that it only works for the cluster with autostop/down setup or the spot cluster, as the normal cluster will not trigger the refresh. If a normal cluster is terminated in the console manually, the skypilot will still try to ssh into it and timeout after a while.

[concretevitamin]

Nice, thanks @Michaelvll!

[concretevitamin]

One potential issue -- I had two clusters, dbg and spot controller. The former was manually killed in console. Then,

» sky queue
Fetching and parsing job queue...
ValueError: Cluster 'dbg' not found on the cloud provider. It was either preempted, autodowned, or manually terminated in console.

» sky queue                     1 ↵
Fetching and parsing job queue...
Failed to get the job queue for cluster 'sky-spot-controller-8a3968f2'.
  sky.exceptions.ClusterNotUpError: Getting the job queue: skipped for cluster 'sky-spot-controller-8a3968f2' (status: STOPPED). It is only allowed for UP clusters.

A multi-cluster command like sky queue raised in the middle rather than processing all clusters. Should we / how should we handle this?

[Michaelvll]

One potential issue -- I had two clusters, dbg and spot controller. The former was manually killed in console. Then,

» sky queue
Fetching and parsing job queue...
ValueError: Cluster 'dbg' not found on the cloud provider. It was either preempted, autodowned, or manually terminated in console.

» sky queue                     1 ↵
Fetching and parsing job queue...
Failed to get the job queue for cluster 'sky-spot-controller-8a3968f2'.
  sky.exceptions.ClusterNotUpError: Getting the job queue: skipped for cluster 'sky-spot-controller-8a3968f2' (status: STOPPED). It is only allowed for UP clusters.

A multi-cluster command like sky queue raised in the middle rather than processing all clusters. Should we / how should we handle this?

Thanks for catching this @concretevitamin! We previously forgot to catch the ValueError, which causes the problem. It should now be fixed. : )