[rllib] Add the ability to run arbitrary Python scripts with ray.tune #1132

ericl · 2017-10-16T06:02:57Z

This builds on top of #1107 to also support evaluation of standalone Python scripts.

See the README for a detailed example of parallelizing evaluation of a MNIST script.

I also did some refactoring to move common classes from RLlib into the ray.tune dir, however this isn't completely finished so ray.tune still has some RLlib dependencies.

cc @richardliaw @pcmoritz

ericl · 2017-10-16T06:03:46Z

python/ray/rllib/tuned_examples/hopper-ppo.yaml

@@ -5,4 +5,4 @@ hopper-ppo:
    resources:
       cpu: 64
       gpu: 4
-    config: {"gamma": 0.995, "kl_coeff": 1.0, "num_sgd_iter": 20, "sgd_stepsize": 1e-4, "sgd_batchsize": 32768, "devices": ["/gpu:0", "/gpu:1", "/gpu:2", "/gpu:3"], "tf_session_args": {"device_count": {"GPU": 4}, "log_device_placement": false, "allow_soft_placement": true}, "timesteps_per_batch": 160000, "num_workers": 64}
+    config: {"gamma": 0.995, "kl_coeff": 1.0, "num_sgd_iter": 20, "sgd_stepsize": .0001, "sgd_batchsize": 32768, "devices": ["/gpu:0", "/gpu:1", "/gpu:2", "/gpu:3"], "tf_session_args": {"device_count": {"GPU": 4}, "log_device_placement": false, "allow_soft_placement": true}, "timesteps_per_batch": 160000, "num_workers": 64}


Apparently, pyyaml has a bug where it can't parse scientific notation correctly.

ericl · 2017-10-16T06:04:11Z

python/ray/tune/examples/tune_mnist_ray.py

+                x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))
+
+
+def train(config={'activation': 'relu'}, reporter=None):


This is the integration with ray.tune

AmplabJenkins · 2017-10-16T06:40:43Z

Merged build finished. Test FAILed.

AmplabJenkins · 2017-10-16T06:40:43Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2142/
Test FAILed.

AmplabJenkins · 2017-10-16T09:27:45Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2145/
Test FAILed.

pcmoritz · 2017-10-16T19:49:57Z

Thanks, this is really useful (most people have standalone training scripts); will review it later today!

AmplabJenkins · 2017-10-16T20:06:22Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-10-16T20:06:23Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2146/
Test PASSed.

richardliaw

Some comments.

richardliaw · 2017-10-16T20:16:53Z

python/ray/tune/trial.py

-    RUNNING = 'RUNNING'
-    TERMINATED = 'TERMINATED'
-    ERROR = 'ERROR'
+    PENDING = "PENDING"


what is the proper format for strings?

I think we prefer double quotes.

This is fine, we've mostly been using double quotes.

richardliaw · 2017-10-16T20:33:57Z

python/ray/tune/trial.py

@@ -98,7 +97,7 @@ def stop(self, error=False):

        try:
            if self.agent:
-                self.agent.stop.remote()
+                ray.get(self.agent.stop.remote())


are there any tradeoffs to doing this?

Hmm maybe it shouldn't block, I removed this.

richardliaw · 2017-10-16T20:47:24Z

python/ray/tune/trial.py

-            self.last_result.training_iteration,
-            round(self.last_result.episode_reward_mean, 1))
+
+        pieces = [


this is nice - could use a dict that maps from formatting string to result and it would probably be cleaner.

richardliaw · 2017-10-16T20:58:49Z

python/ray/rllib/train.py

@@ -32,16 +32,20 @@
 # Extends the base parser defined in ray/tune/config_parser, to add some
 # RLlib specific arguments. For more arguments, see the configuration
 # defined there.
-parser.add_argument("--redis-address", default=None, type=str,
+parser.add_argument("--redis_address", default=None, type=str,


http://ray.readthedocs.io/en/latest/using-ray-on-a-cluster.html has redis-address, makes sense to standardize.

richardliaw · 2017-10-16T21:01:31Z

python/ray/tune/README.rst

+finer-grained control over trial setup and scheduling. Some examples of
+calling ray.tune programmatically include:
+
+- python/ray/tune/examples/tune_mnist_ray.py


python/ray/tune/examples/tune_mnist_ray.py

(formatting `` would be nice)

richardliaw · 2017-10-16T21:33:13Z

python/ray/tune/script_runner.py

+        elif hasattr(importlib, "machinery"):
+            # Python 3.3
+            from importlib.machinery import SourceFileLoader
+            foo = SourceFileLoader("external_file", file_path).load_module()


nit: foo...

richardliaw · 2017-10-16T23:42:10Z

python/ray/tune/examples/tune_mnist_ray.py

+            if i % 10 == 0:
+                train_accuracy = accuracy.eval(feed_dict={
+                        x: batch[0], y_: batch[1], keep_prob: 1.0})
+                if status_reporter:


Would be nice to annotate this part to indicate specific modification of status reporter here.

AmplabJenkins · 2017-10-17T00:17:29Z

Merged build finished. Test FAILed.

AmplabJenkins · 2017-10-17T00:17:30Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2150/
Test FAILed.

AmplabJenkins · 2017-10-17T00:20:29Z

Merged build finished. Test FAILed.

AmplabJenkins · 2017-10-17T00:20:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2151/
Test FAILed.

AmplabJenkins · 2017-10-17T03:15:16Z

Merged build finished. Test FAILed.

AmplabJenkins · 2017-10-17T03:15:17Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2152/
Test FAILed.

AmplabJenkins · 2017-10-17T08:18:12Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-10-17T08:18:13Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2153/
Test PASSed.

AmplabJenkins · 2017-10-17T20:31:04Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-10-17T20:31:05Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2154/
Test PASSed.

ericl · 2017-10-18T08:12:35Z

I've seen this a few times in the failing tests cc @stephanie-wang


testManyLocalSchedulersDying (__main__.ActorReconstruction) ... Waiting for redis server at 127.0.0.1:22195 to respond...

Traceback (most recent call last):

  File "/Users/travis/.local/lib/python2.7/site-packages/ray-0.2.1-py2.7-macosx-10.7-x86_64.egg/ray/workers/default_worker.py", line 94, in <module>

    ray.worker.global_worker.main_loop()

  File "/Users/travis/.local/lib/python2.7/site-packages/ray-0.2.1-py2.7-macosx-10.7-x86_64.egg/ray/worker.py", line 858, in main_loop

    self._wait_for_and_process_task(task)

  File "/Users/travis/.local/lib/python2.7/site-packages/ray-0.2.1-py2.7-macosx-10.7-x86_64.egg/ray/worker.py", line 808, in _wait_for_and_process_task

    self._process_task(task)

  File "/Users/travis/.local/lib/python2.7/site-packages/ray-0.2.1-py2.7-macosx-10.7-x86_64.egg/ray/worker.py", line 747, in _process_task

    e, traceback_str)

  File "/Users/travis/.local/lib/python2.7/site-packages/ray-0.2.1-py2.7-macosx-10.7-x86_64.egg/ray/worker.py", line 772, in _handle_process_task_failure

    self._store_outputs_in_objstore(return_object_ids, failure_objects)

  File "/Users/travis/.local/lib/python2.7/site-packages/ray-0.2.1-py2.7-macosx-10.7-x86_64.egg/ray/worker.py", line 686, in _store_outputs_in_objstore

    self.put_object(objectids[i], outputs[i])

  File "/Users/travis/.local/lib/python2.7/site-packages/ray-0.2.1-py2.7-macosx-10.7-x86_64.egg/ray/worker.py", line 339, in put_object

    self.store_and_register(object_id, value)

  File "/Users/travis/.local/lib/python2.7/site-packages/ray-0.2.1-py2.7-macosx-10.7-x86_64.egg/ray/worker.py", line 292, in store_and_register

    object_id.id()), self.serialization_context)

  File "plasma.pyx", line 395, in pyarrow.plasma.PlasmaClient.put

  File "plasma.pyx", line 301, in pyarrow.plasma.PlasmaClient.create

  File "error.pxi", line 79, in pyarrow.lib.check_status

ArrowIOError: Broken pipe

robertnishihara · 2017-10-18T17:43:47Z

Yeah, test failures all look innocuous to me.

robertnishihara · 2017-10-18T17:44:08Z

I'd suggest merging unless there are other changes you want to do here.

ericl added 26 commits October 13, 2017 16:41

fix yaml bug

07e9263

add ext agent

ed9d2d1

gpus

4f4d6de

update

2aaf661

tuning

bc86b46

docs

96775f1

Sun Oct 15 21:09:25 PDT 2017

56ad4ea

lint

55b2386

update

a9e8345

Sun Oct 15 22:39:55 PDT 2017

140b612

Sun Oct 15 22:40:17 PDT 2017

924741d

Sun Oct 15 22:43:06 PDT 2017

9524e03

Sun Oct 15 22:46:06 PDT 2017

d74fafc

Sun Oct 15 22:46:21 PDT 2017

c0a98b1

Sun Oct 15 22:48:11 PDT 2017

2f5a93f

Sun Oct 15 22:48:44 PDT 2017

de3f5f9

Sun Oct 15 22:49:23 PDT 2017

9cea84e

Sun Oct 15 22:50:21 PDT 2017

a5eb3bb

Sun Oct 15 22:53:00 PDT 2017

be304f8

Sun Oct 15 22:53:34 PDT 2017

da8fff1

Sun Oct 15 22:54:33 PDT 2017

f3b75b7

Sun Oct 15 22:54:50 PDT 2017

f7ba39f

Sun Oct 15 22:55:20 PDT 2017

b3ae696

Sun Oct 15 22:56:56 PDT 2017

2eedda8

Sun Oct 15 22:59:03 PDT 2017

7477228

fix

2c6ccd0

ericl commented Oct 16, 2017

View reviewed changes

ericl added 3 commits October 16, 2017 12:15

fix ex

ef8f5c0

py2 support

88c3a96

upd

d8babc2

richardliaw reviewed Oct 16, 2017

View reviewed changes

comments

1060ca9

richardliaw reviewed Oct 16, 2017

View reviewed changes

ericl added 2 commits October 16, 2017 16:49

comments

2b64a06

cleanup readme

0d4d6c6

fix trial

9cea4cf

annotate

0323c24

pcmoritz approved these changes Oct 17, 2017

View reviewed changes

Update rllib.rst

d5ba6da

richardliaw merged commit 5a50e0e into ray-project:master Oct 18, 2017

		x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))


		def train(config={'activation': 'relu'}, reporter=None):

[rllib] Add the ability to run arbitrary Python scripts with ray.tune #1132

[rllib] Add the ability to run arbitrary Python scripts with ray.tune #1132

Conversation

ericl commented Oct 16, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Oct 16, 2017

AmplabJenkins commented Oct 16, 2017

AmplabJenkins commented Oct 16, 2017

pcmoritz commented Oct 16, 2017

AmplabJenkins commented Oct 16, 2017

AmplabJenkins commented Oct 16, 2017

richardliaw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Oct 17, 2017

AmplabJenkins commented Oct 17, 2017

AmplabJenkins commented Oct 17, 2017

AmplabJenkins commented Oct 17, 2017

AmplabJenkins commented Oct 17, 2017

AmplabJenkins commented Oct 17, 2017

AmplabJenkins commented Oct 17, 2017

AmplabJenkins commented Oct 17, 2017

AmplabJenkins commented Oct 17, 2017

AmplabJenkins commented Oct 17, 2017

ericl commented Oct 18, 2017

robertnishihara commented Oct 18, 2017

robertnishihara commented Oct 18, 2017