[Core] Ray Label Selector API Implementation Tracker

[ryanaoleary]

Description

Implement support for label selector API:

• (P0) Ray saves label info associated with a node in GcsNodeInfo - already implemented
• (P0) Update --labels argument to take either a list of strings or read from file and expose this API publicly
  • Update --labels and add --labels-file options for Label Selector API #51706
• (P0) Add label_selector API to @ray.remote decorator to schedule tasks/actors
  • Add label_selector option to remote functions #51707
• (P0) Update ClusterResourceScheduler::GetBestSchedulableNode to enforce label_selector conditions when returning list of candidate nodes. This will eventually replace SchedulingOptions::NodeLabelScheduling(scheduling_strategy).
  • Add label selector scheduling logic to Cluster Resource Manager #51901
  • [Core] Remove RAY_LOG(ERROR from LabelSelector and validate in prepare_label_selector #52964
• (P1) Add node labels to runtime context for tasks/actors
  • [Core] Add node labels to runtime context #52926
• (P1) Add bundle_label_selector to the ray.util.placement_group constructor to apply a set of label_selectors to placement group bundles
  • [Core] Add bundle_label_selector scheduling logic #52988
• (P0) Populate list of default labels automatically, currently only supports ray.io/node-id, from K8s [Core] Add default Ray Node labels at Node init #53360

Autoscaler adaptation:

• (P1) Update Autoscaler data model to pass label information by adding a labels field to the ResourceRequest message
  • [Core][Autoscaler] Update the Autoscaler Resource Requests Data Model for Scheduling for Label Selector #51771
• (P1) Adapt Ray V2 Autoscaler to parse labels from K8s Pod Spec and generate a --labels arg to rayStartParams and potential code cleanup.
  • Done by generating default labels from K8s Pod spec in KubeRay: Add default Ray node label info to Ray Pod environment kuberay#3699
  • Add top-level Labels and Resources Structed fields to HeadGroupSpec and WorkerGroupSpec kuberay#4106
• (P1) Update Autoscaler bin packing logic to directly consider label matching
  • [Core] Update V2 Autoscaler to support scheduling using Node labels and LabelSelector API #53578
• (P1) Update the Autoscaler code path to handle the label information passed back from GCS
  • [Core] Update V2 Autoscaler to support scheduling using Node labels and LabelSelector API #53578
• (P1) Add labels to autoscaling config by parsing rayStartParams
  • [Core][Autoscaler] Add labels to KubeRay autoscaling config #56532
  • Changed to use top-level field: [Autoscaler][V2] Add top-level Resources and Labels field to KubeRay Autoscaling config #57260
• (P1) Add parsing labels in other types of cluster launchers
• (P2) Cleanup default labels dead code

Documentation/Library changes

• (P1) End-to-end test for label based scheduling milestone 1
  • [Autoscaler][V2] Consider bundle_label_selector in Ray V2 Autoscaler #56826
• (P1) End-to-end example in Kuberay doc for label selector with autoscaling
• (P1) Update documentation/examples to use updated label_selector API for non-fallback use cases
  • [DOC-127] MVP for OSS Ray labels #54254
• (P1) Replacing the library usage of NodeLabelSchedulingStrategy(soft=false) with label based scheduling
  • [Core] Update cluster scheduler to handle label selector hard node id constraint #56235
  • Replace NodeAffinitySchedulingStrategy when soft=False with Label Selector API #54940
• (P1) Support passing labels from head and worker group specs in RayCluster CR in KubeRay to Ray nodes
• (P1) Add label selector dimension into the infeasible task cancellation mechanism
• (P2) Add labels argument to request_resource() SDK function used by Ray libraries
  • [Autoscaler] Add bundle_label_selector to request_resources sdk #54843
• (P2) Determine whitelist of K8s labels to always pass to Ray nodes
• (P2) Add required_labels to TaskState schema to expose labels in state API
  • Add observability for label-selectors #53423
• (P2) Add node labels & task/actor label selectors to Ray Events
• (P2) Ray libraries leverages the new label selector API
  • [Serve] Add label_selector and bundle_label_selector to Serve API #57694

Milestone 2:

• (P0) Update the label_selector API in @ray.remote decorator to support label fallback syntax
  • [Core] Add fallback_strategy API to Task/Actor remote options #56374
• (P0) Update the bundle_label_selector in the ray.util.placement_group constructor to support label fallback syntax
• (P0) Implement fallback_strategy scheduling logic for tasks, actors and placement groups
  • [Core] Add fallback strategy scheduling logic #56369
• (P0) Update the gcs logic to pass the fallback strategies to autoscaler
• (P0) Update Autoscaler bin packing logic to support label fallback syntax
• (P0) Add exists() and !exists() label operators in label selectors
• (P0) Add validation to prevent empty string value in node labels & label selectors in Ray code, Autoscaler and Kuberay
• (P1) Update documentation/examples to use updated label_selector API for label fallback use cases
• (P2) Update library usage of NodeLabelSchedulingStrategy, _soft_target_node_id and other related features with label_selector API
• (P2) Add deprecation warnings for the NodeLabelSchedulingStrategy , accelerator types and other features that will be replaced by label based scheduling

Use case

This issue will serve to track the progress of implementing the label selector API feature enhancement. This enhancement supersedes the previous node affinity feature enhancement REP and continues on much of the implementation there.

Node affinity feature enhancement work tracker: #34894
Related REP: ray-project/enhancements#60

[Moonquakes]

Hi @ryanaoleary , can we support dynamic modification of node labels? There is a useful scenario: there are a large number of small tasks and a small number of long-tail tasks. If these long-tail tasks are scheduled by default, they are likely to be distributed on different nodes. If we can add labels like "long-time" to these long-tail tasks and then schedule them to a few nodes, we can effectively reduce resource waste.

However, if we want to implement this function, we need to be able to dynamically modify the node labels so that the nodes assigned to long-tail tasks can only schedule long-tail tasks in the future. Otherwise, we need to set two different labels for each type of node in advance, which is not very user-friendly.

[MengjinYan]

Thanks for the feedback and context, @Moonquakes!

Currently, updating the node labels is not in our plan yet. At the same time, we are open to supporting it in later milestones after further evaluation considering the use cases.

[ryanaoleary]

I think the minimal tasks we'll need to complete to support a demo on a static RayCluster are:

• Update --labels argument to take either a list of strings or read from file and expose this API publicly
• Add label_selector API to @ray.remote decorator to schedule tasks/actors
• Populate list of default labels automatically - specifically from K8s
• Update ClusterResourceScheduler::GetBestSchedulableNode to enforce label_selector conditions when returning list of candidate nodes.

And if we're using Ray Serve,

• Update Ray Serve RayActorOptionsSchema with label_selector API options

I'll mark these tasks as P0s.