[Umbrella] GCS fault tolerance on KubeRay

[kevin85421]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Ray

• [Dashboard][Observability] Dashboard shows that a task is still running after a RayCluster with GCS FT restarts  ray#34507
• [Core][GCS FT] Two alive head nodes will be present after the original head node recovers from a failure ray#39122
• [Doc][GCS FT] A Kubernetes Job to clean up Redis cache #1424
• [Bug] Deleting RayService does not clear Redis cache #1286 (comment)
  • [Feature] Automatically expire GCS FT Redis keys #959

KubeRay

• [Bug][GCS FT] Worker pods crash unexpectedly when gcs_server on head pod is killed #1036
• [Bug] Deleting RayService does not clear Redis cache #1286
• [Bug] Adding GCS FT breaks incremental RayService deployments #1296
• [Bug] [RayService] Cannot move from single app to multi-app without downtime if using GCS FT #1297 [WIP]
• [GCS FT][Refactor] Redefine the behavior for deleting Pods and stop listening to Kubernetes events #1341
• [Feature] How should we handle PodFailed Pods? #1144
• [GCS FT] end-to-end tests for #1341 #1353
• [GCS FT] Give readiness / liveness probes good default values #1354
• [GCS FT] Consider the case of sidecar containers #1355
• [Feature][GCS FT] Redis cleanup improvement #1557
• Fault Tolerance not working with external redis  ray#40562

• Under discussions
  • [Discussion][GCS FT] Delete Pods by the value of restartCount #1356

KubeRay tests / docs

• [Feature] e2e tests for RayService high-availability without GCS FT #1421
• [Feature][GCS FT] Test the behavior of cleanup_redis_storage #1422
• [Doc] Create a doc for #1386  #1393

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!