[xray] raylet scheduling mechanism with a simple spillback policy #2749

atumanov · 2018-08-27T20:19:05Z

What do these changes do?

distribute load and resource information on a heartbeat
for each raylet, maintain total and available resource capacity as well as measure of current load
this PR introduces a new notion of load, defined as a sum of all resource demand induced by queued ready tasks on the local raylet. This provides a heterogeneity-aware measure of load that supersedes legacy Ray's task count as a proxy for load.
modify the scheduling policy to perform capacity-based, load-aware, optimistically concurrent resource allocation
perform task spillover to the heartbeating node in response to a heartbeat, implementing heterogeneity-aware late-binding/work-stealing.

…e worker is blocked

…at to hex

AmplabJenkins · 2018-08-27T21:28:02Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7787/
Test FAILed.

…to raylet-scheduling-simple

AmplabJenkins · 2018-08-27T21:55:08Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7790/
Test FAILed.

AmplabJenkins · 2018-08-27T22:27:03Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7789/
Test PASSed.

AmplabJenkins · 2018-08-27T23:09:09Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7791/
Test PASSed.

AmplabJenkins · 2018-08-27T23:28:52Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7792/
Test PASSed.

…uarantee object eviction.

robertnishihara · 2018-08-28T01:37:55Z

src/ray/raylet/scheduling_resources.h

+  /// resource labels to this set.
+  ///
+  /// \param other: The other resource set to add.
+  void AddResources(const ResourceSet &other);


\return Void.

robertnishihara · 2018-08-28T01:39:10Z

src/ray/raylet/scheduling_policy.h

+  /// information for some subset of the cluster. For all client IDs in the returned
+  /// placement map, the corresponding SchedulingResources::resources_load_ is
+  /// incremented by the aggregate resource demand of the tasks assigned to it.
+  ///


there shouldn't be an extra /// here

robertnishihara · 2018-08-28T01:39:16Z

src/ray/raylet/scheduling_policy.h

-      const std::unordered_map<ClientID, SchedulingResources> &cluster_resources,
-      const ClientID &local_client_id, const std::vector<ClientID> &others);
+      std::unordered_map<ClientID, SchedulingResources> &cluster_resources,
+      const ClientID &local_client_id);


document local_client_id

robertnishihara · 2018-08-28T01:40:57Z

src/ray/raylet/node_manager.cc

+    std::unordered_map<ClientID, SchedulingResources> local_resource_map(
+        {{local_client_id, cluster_resource_map_[local_client_id]}});
+    // Invoke the scheduling policy only on local resources.
+    // ScheduleTasks(local_resource_map);


Remove this and the comment

robertnishihara · 2018-08-28T01:54:31Z

We should also update the diagram in https://github.com/ray-project/ray/blob/master/src/ray/raylet/design_docs/task_states.rst.

AmplabJenkins · 2018-08-28T02:49:09Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7806/
Test PASSed.

AmplabJenkins · 2018-08-28T05:29:28Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7812/
Test FAILed.

atumanov · 2018-08-28T05:53:52Z

Jenkins retest this please

AmplabJenkins · 2018-08-28T06:08:10Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7814/
Test PASSed.

AmplabJenkins · 2018-08-28T06:44:10Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7816/
Test FAILed.

AmplabJenkins · 2018-08-28T07:08:37Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7818/
Test PASSed.

atumanov and others added 30 commits August 25, 2018 14:53

raylet scheduling mechanism, load accounting/distribution, policy

c28452d

fix upstream/master rebase

abe0346

invoke scheduling policy on local resources prior to dispatch when th…

1af4188

…e worker is blocked

addressing comments

2e424bb

schedule before dispatch

c719949

enable all ActorsWithGPUs tests

ad2206e

remove excessive loggingj

f0683d0

addressing comments: minor resource set API refactor

ac51981

Call DispatchTasks after ScheduleTasks.

5b6bba1

Another DispatchTasks after ScheduleTasks.

d3d0ca9

addressing comments

dac1ad8

monitor heartbeat handler updated to decode clientid from binary form…

e9b4202

…at to hex

making load balancing tasks with dependencies sleep for 10ms

4f82d6a

Make tests slightly more verbose so we know which test is hanging.

96fc26f

Fix linting.

afa4a88

Dispatch tasks when new worker becomes available.

0a46231

Add code for converting scheduling queues to human-readable string.

7bdf6c7

Remove outdated Java test UIDTest.

cd4a3bd

Fix linting.

c3e70f6

handle placeable -> waiting with remaining placeable

3610946

linting

6812180

scheduling all placeable tasks; modified heartbeat to spillover ready

053c5e5

eliminate do/while around scheduling policy invocation

0adb158

remove some calls to the scheduling policy

df6d71d

don't spill over actor tasks from the ready queue

b40c7f9

handle infeasible tasks

fc2c788

handle infeasible tasks

d5b8df9

move infeasible tasks from placeable to infeasible

3be0c65

hastask should check infeasible

6bd29d2

Dispatch tasks if we fail to write assign a task to a worker.

ab7f67a

Linting.

afc4940

robertnishihara and others added 4 commits August 27, 2018 14:46

Doc and linting.

290aebf

check for hard constraints when spilling ready tasks

430d389

Merge branch 'raylet-scheduling-simple' of github.com:atumanov/ray in…

c6134b3

…to raylet-scheduling-simple

Linting

1ab3fad

Skip testActorMultipleGPUsFromMultipleTasks in actor_test.py.

faa279b

Remove multinode test_driver_put_errors because it doesn't actually g…

d8078b6

…uarantee object eviction.

robertnishihara reviewed Aug 28, 2018

View reviewed changes

robertnishihara added 2 commits August 27, 2018 21:15

Minor improvements

26ae1ae

Update ascii art scheduling state diagram.

aec71d5

robertnishihara force-pushed the raylet-scheduling-simple branch from bfa4406 to aec71d5 Compare August 28, 2018 04:52

Update doc

079c723

robertnishihara approved these changes Aug 28, 2018

View reviewed changes

robertnishihara merged commit de047da into ray-project:master Aug 28, 2018

robertnishihara deleted the raylet-scheduling-simple branch August 28, 2018 07:03

This was referenced Aug 28, 2018

Bump version to 0.5.1. #2755

Merged

[tune] Request for a temporary releasing tag #2689

Closed

atumanov mentioned this pull request Aug 30, 2018

[xray] raylet scheduling mechanism #2420

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[xray] raylet scheduling mechanism with a simple spillback policy #2749

[xray] raylet scheduling mechanism with a simple spillback policy #2749

atumanov commented Aug 27, 2018

AmplabJenkins commented Aug 27, 2018

AmplabJenkins commented Aug 27, 2018

AmplabJenkins commented Aug 27, 2018

AmplabJenkins commented Aug 27, 2018

AmplabJenkins commented Aug 27, 2018

robertnishihara Aug 28, 2018

robertnishihara Aug 28, 2018

robertnishihara Aug 28, 2018

robertnishihara Aug 28, 2018

robertnishihara commented Aug 28, 2018

AmplabJenkins commented Aug 28, 2018

AmplabJenkins commented Aug 28, 2018

atumanov commented Aug 28, 2018

AmplabJenkins commented Aug 28, 2018

AmplabJenkins commented Aug 28, 2018

AmplabJenkins commented Aug 28, 2018

[xray] raylet scheduling mechanism with a simple spillback policy #2749

[xray] raylet scheduling mechanism with a simple spillback policy #2749

Conversation

atumanov commented Aug 27, 2018

What do these changes do?

AmplabJenkins commented Aug 27, 2018

AmplabJenkins commented Aug 27, 2018

AmplabJenkins commented Aug 27, 2018

AmplabJenkins commented Aug 27, 2018

AmplabJenkins commented Aug 27, 2018

robertnishihara Aug 28, 2018

Choose a reason for hiding this comment

robertnishihara Aug 28, 2018

Choose a reason for hiding this comment

robertnishihara Aug 28, 2018

Choose a reason for hiding this comment

robertnishihara Aug 28, 2018

Choose a reason for hiding this comment

robertnishihara commented Aug 28, 2018

AmplabJenkins commented Aug 28, 2018

AmplabJenkins commented Aug 28, 2018

atumanov commented Aug 28, 2018

AmplabJenkins commented Aug 28, 2018

AmplabJenkins commented Aug 28, 2018

AmplabJenkins commented Aug 28, 2018