[Azure] KeyError when launching Azure cluster

[Michaelvll]

The following error occurs occasionally. It seems to happen more often when test_cancel_azure and test_azure_start_stop run together.

  File "/home/ubuntu/skypilot/sky/skylet/providers/azure/node_provider.py", line 362, in _get_cached_node
    return self._get_node(node_id=node_id)
  File "/home/ubuntu/skypilot/sky/skylet/providers/azure/node_provider.py", line 357, in _get_node
    return self.cached_nodes[node_id]
KeyError: 'ray-test-cancel-azure-3446f302-2b-head-138eaa0b0'

@suquark Do you think there is a race condition around the following place? Should it be a file lock across multiple ray up?

skypilot/sky/skylet/providers/azure/node_provider.py

Lines 79 to 80 in 21009ef

	@synchronized
	def _get_filtered_nodes(self, tag_filters):

[Michaelvll]

This happens much more often recently. Do we have any idea @suquark ? :)

[Michaelvll]

It seems the problem happens when the test tries to sky start the cluster is just stopped with sky stop.

[github-actions]

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[Michaelvll]

This has not been observed for a while. Closed for now.