Cluster Utilities for Fault Tolerance Tests #3008

richardliaw · 2018-10-02T02:03:41Z

Adds some utilities for creating and killing nodes in a cluster. Useful for fault tolerance tests.

TODOs:

Add basic tests for verifying functionality
- Add kill/add node test
Documentation
Write a duplicate test for component_failure_test.py
Add sane defaults

AmplabJenkins · 2018-10-02T03:30:33Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8507/
Test PASSed.

robertnishihara · 2018-10-03T18:02:37Z

Related to #609.

richardliaw · 2018-10-06T00:07:39Z

python/ray/test/cluster_utils.py

+    # Log monitor doesn't die for some reason
+    worker.kill_log_monitor()
+    worker.kill_plasma_store()
+    # TODO(rliaw): how to wait for raylet timeout?


@robertnishihara any idea for this?

AmplabJenkins · 2018-10-06T00:57:01Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8562/
Test FAILed.

AmplabJenkins · 2018-10-06T01:26:50Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8565/
Test PASSed.

AmplabJenkins · 2018-10-18T01:56:40Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8711/
Test FAILed.

AmplabJenkins · 2018-10-18T02:16:07Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8713/
Test PASSed.

AmplabJenkins · 2018-10-18T02:39:14Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8714/
Test PASSed.

AmplabJenkins · 2018-10-18T03:35:19Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8716/
Test PASSed.

robertnishihara · 2018-10-18T06:48:23Z

python/ray/test/cluster_utils.py

+logger = logging.getLogger(__name__)
+
+
+class Cluster():


Cluster(object) to support Python 2

robertnishihara · 2018-10-18T06:48:45Z

python/ray/test/cluster_utils.py

+    def __init__(self, initialize_head=False, connect=False, **head_node_args):
+        """Initializes the cluster.
+
+        Arguments:


robertnishihara · 2018-10-18T06:49:23Z

python/ray/test/cluster_utils.py

+        """
+        self.head_node = None
+        self.worker_nodes = {}
+        self.redis_address = ""


I think None would be better than ""

robertnishihara · 2018-10-18T06:50:04Z

python/ray/test/cluster_utils.py

+            self.add_node(**head_node_args)
+            if connect:
+                ray.init(redis_address=self.redis_address)
+        elif connect:


Maybe just raise an exception earlier on if not initialize_head and connect

robertnishihara · 2018-10-18T06:51:12Z

python/ray/test/cluster_utils.py

+        self.process_dict["log_monitor"][0].kill()
+        self.process_dict["log_monitor"][0].wait()
+
+    def kill_allprocesses(self):


kill_all_processes

robertnishihara · 2018-10-18T06:51:23Z

python/ray/test/cluster_utils.py

+        dies before worker nodes die?
+
+        Returns:
+            List of all nodes, including the head node."""


""" on the next line

robertnishihara · 2018-10-18T06:52:12Z

python/ray/test/cluster_utils.py

+
+    def wait_for_nodes(self):
+        """Waits for all nodes to be registered with global state."""
+        try:


What's the point of this try/except?

If you want to avoid a bad error message when this is called before ray.init, you can check ray.is_connected

robertnishihara · 2018-10-18T06:53:43Z

python/ray/test/cluster_utils.py

+        self.remove_node(self.head_node)
+
+
+class Node():


Node(object)

robertnishihara · 2018-10-18T06:54:11Z

test/multi_node_test_2.py

+    worker.kill_plasma_store()
+    worker.process_dict[services.PROCESS_TYPE_RAYLET][0].wait()
+    assert not worker.any_processes_alive(), worker.live_processes()
+    print("Success")


remove this line

test/multi_node_test_2.py

robertnishihara · 2018-10-18T06:57:40Z

test/multi_node_test_2.py

+    worker2 = g.add_node()
+    g.wait_for_nodes()
+    g.remove_node(worker2)
+    g.wait_for_nodes()


I see, so this is actually waiting for the node to be removed from the client table, right? I guess that's a good thing to test, but unfortunately it takes 10s for that to happen, per call, so I'm guessing these tests are really slow? This might be too much of a burden on our CI...

So this test takes about 15 seconds to run... is there someway we can test this (maybe reduce the 10s wait?)

robertnishihara · 2018-10-18T07:00:49Z

python/ray/test/cluster_utils.py

+    """Abstraction for a Ray node."""
+
+    def __init__(self, process_dict):
+        # TODO(rliaw): Is there a unique identifier for a node?


Each node has a client_id in the client table. What kind of identifier are you looking for?

Yeah; something like that (this is a non-blocking issue though). What would be a way to get the client_id without first calling ray.init? Ideally, it would be part of the output of services.start_ray_node.

robertnishihara · 2018-10-18T07:02:18Z

test/multi_node_test_2.py

+    g.remove_node(worker2)
+    g.wait_for_nodes()
+    ray.shutdown()
+    g.shutdown()


ray.shutdown and g.shutdown are typically the kind of thing you would put in a pytest fixture (after yield) so that they always get called in the event of a test failure. Not sure if it makes sense here, but something to consider..

robertnishihara · 2018-10-18T07:02:34Z

test/multi_node_test_2.py

+    worker.process_dict[services.PROCESS_TYPE_RAYLET][0].wait()
+    assert not worker.any_processes_alive(), worker.live_processes()
+    print("Success")
+    g.shutdown()


You need a ray.shutdown in this test also, right?

AmplabJenkins · 2018-10-19T00:25:40Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8741/
Test FAILed.

robertnishihara · 2018-10-19T04:40:51Z

@richardliaw it looks like you're running the test for legacy Ray as well as xray. You probably want to remove it from the legacy Ray test suite.

richardliaw · 2018-10-19T05:49:09Z

Done.

AmplabJenkins · 2018-10-19T06:44:09Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8749/
Test PASSed.

AmplabJenkins · 2018-10-19T08:21:46Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8753/
Test PASSed.

richardliaw · 2018-10-21T01:55:24Z

@robertnishihara tests passing

robertnishihara · 2018-10-21T05:56:38Z

Thanks @richardliaw!

add cluster utils

05ab9d6

richardliaw self-assigned this Oct 3, 2018

This was referenced Oct 3, 2018

[wip] Client Table Multi-Node Tests #2893

Closed

[tune/core] Use Global State API for resources #3004

Merged

richardliaw added 2 commits October 5, 2018 16:33

Add docs

0fde0f9

add utility for cluster killing

10f0143

richardliaw commented Oct 6, 2018

View reviewed changes

richardliaw added 5 commits October 17, 2018 17:24

Merge branch 'master' into cluster_testing

aa43a2d

Merge branch 'master' into cluster_testing

b4004e6

fix up this test and remove redis setting

d5c4c83

fix up doc

479d88e

add test

10e622d

richardliaw changed the title ~~[wip] Cluster Utilities for Fault Tolerance Tests~~ Cluster Utilities for Fault Tolerance Tests Oct 18, 2018

richardliaw requested a review from robertnishihara October 18, 2018 04:50

robertnishihara reviewed Oct 18, 2018

View reviewed changes

python/ray/test/cluster_utils.py Outdated

self.remove_node(self.head_node)

class Node():

Copy link

Collaborator

robertnishihara Oct 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Node(object)

robertnishihara reviewed Oct 18, 2018

View reviewed changes

test/multi_node_test_2.py Show resolved Hide resolved

robertnishihara reviewed Oct 18, 2018

View reviewed changes

richardliaw added 2 commits October 18, 2018 15:55

fix cluster utils issues

9abfeb2

fix comments for tests

289a005

limit

0ffaa3e

Update multi_node_test_2.py

598499b

robertnishihara mentioned this pull request Oct 20, 2018

[xray] Implement actor reconstruction. #3063

Closed

4 tasks

robertnishihara approved these changes Oct 21, 2018

View reviewed changes

robertnishihara merged commit 40c4148 into ray-project:master Oct 21, 2018

robertnishihara deleted the cluster_testing branch October 21, 2018 05:56

Uh oh!

Cluster Utilities for Fault Tolerance Tests #3008

Cluster Utilities for Fault Tolerance Tests #3008

Uh oh!

Conversation

richardliaw commented Oct 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AmplabJenkins commented Oct 2, 2018

Uh oh!

robertnishihara commented Oct 3, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Oct 6, 2018

Uh oh!

AmplabJenkins commented Oct 6, 2018

Uh oh!

AmplabJenkins commented Oct 18, 2018

Uh oh!

AmplabJenkins commented Oct 18, 2018

Uh oh!

AmplabJenkins commented Oct 18, 2018

Uh oh!

AmplabJenkins commented Oct 18, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

richardliaw Oct 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Oct 19, 2018

Uh oh!

robertnishihara commented Oct 19, 2018

Uh oh!

richardliaw commented Oct 19, 2018

Uh oh!

AmplabJenkins commented Oct 19, 2018

Uh oh!

AmplabJenkins commented Oct 19, 2018

Uh oh!

richardliaw commented Oct 21, 2018

Uh oh!

robertnishihara commented Oct 21, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

richardliaw commented Oct 2, 2018 •

edited

Loading

richardliaw Oct 18, 2018 •

edited

Loading