[rllib] Initial work on integrating hyperparameter search tool #1107

ericl · 2017-10-11T06:23:46Z

This PR refactors the rllib train.py script to depend on a new ray.tune module, which implements efficient hyper-parameter search.

The overall usage of train.py remains roughly the same, though now it supports two modes:

Inline args:
./train.py --env=Pong-v0 --alg=PPO --num_trials=8 --stop '{"time_total_s": 3200}' --resources '{"cpu": 8, "gpu": 2}' --config '{"num_workers": 8, "sgd_num_iter": 10}'
File-based:
./train.py -f tune-pong.yaml

Both delegate scheduling of trials to the ray.tune TrialRunner class. Additionally, the file-based mode supports hyper-parameter tuning (currently just grid and random search).

Note that though ray.tune should be written to support generic training, right now it has some RL-specific notions like agents and envs, which we should try to remove.

cc @richardliaw @pcmoritz

richardliaw · 2017-10-11T06:26:08Z

python/ray/rllib/a3c/a3c.py

@@ -11,7 +11,7 @@
 import ray
 from ray.rllib.a3c.runner import RunnerThread, process_rollout
 from ray.rllib.a3c.envs import create_and_wrap
-from ray.rllib.common import Agent, TrainingResult, get_tensorflow_log_dir


is this intended?

Oh, seems like Eric split local_log_dir and upload_dir, that's a better way to do it than what we had :)

richardliaw · 2017-10-11T06:29:41Z

python/ray/tune/trial.py

+        return self.agent.train.remote()
+
+    def should_stop(self, result):
+        """Whether the given result meets this trial's stopping criteria."""


In this PR, we only support "maximization" right now?

Yeah, the user can just invert their objective for now

richardliaw · 2017-10-11T06:35:52Z

python/ray/tune/trial_runner.py

+
+        self._trials = []
+        self._pending = {}
+        self._avail_resources = {'cpu': 0, 'gpu': 0}


This dict format appears very often; it could make sense in readability and otherwise to make this into a named tuple "Resources"...

AmplabJenkins · 2017-10-11T06:58:18Z

Merged build finished. Test FAILed.

AmplabJenkins · 2017-10-11T06:58:19Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2085/
Test FAILed.

pcmoritz · 2017-10-11T17:42:24Z

python/ray/rllib/agents.py

+            timesteps_this_iter=10, info={})
+
+
+def get_agent_class(alg):


Oh yeah! Good call on adding this function 👍

pcmoritz

Very cool! Can you add a little bit of documentation on how to use it?

richardliaw · 2017-10-12T01:53:36Z

python/ray/tune/trial.py

+    def train_remote(self):
+        """Returns Ray future for one iteration of training."""
+
+        assert self.status == Trial.RUNNING, self.status


The second status is printed out if the first expression is false?

richardliaw · 2017-10-12T02:02:11Z

python/ray/tune/config_parser.py

+    values, or `eval: <str>` for values to be sampled from the given Python
+    expression.
+
+    See ray/rllib/tuned_examples for more examples of configs in YAML form.


I feel we will easily lose track of this line (especially as we move the example code around), might make sense to move this to a README

along with the example yaml

Added README

richardliaw · 2017-10-12T02:06:54Z

python/ray/tune/trial.py

+        agent_cls = get_agent_class(self.alg)
+        cls = ray.remote(num_gpus=self.resources.get('gpu', 0))(agent_cls)
+        self.agent = cls.remote(
+            self.env_creator, self.config, self.local_dir,


(don't know where to leave this) but just pointing out this does not support S3 directories.

richardliaw · 2017-10-12T02:10:22Z

python/ray/tune/config_parser.py

+                        help="Number of trials to evaluate.")
+    parser.add_argument("--local_dir", default="/tmp/ray", type=str,
+                        help="Local dir to save training results to.")
+    parser.add_argument("--upload_dir", default=None, type=str,


this is unused

ericl

Updated

ericl · 2017-10-12T04:42:53Z

python/ray/tune/config_parser.py

+                        help="Number of trials to evaluate.")
+    parser.add_argument("--local_dir", default="/tmp/ray", type=str,
+                        help="Local dir to save training results to.")
+    parser.add_argument("--upload_dir", default=None, type=str,


ericl · 2017-10-12T04:51:21Z

python/ray/tune/config_parser.py

+    values, or `eval: <str>` for values to be sampled from the given Python
+    expression.
+
+    See ray/rllib/tuned_examples for more examples of configs in YAML form.


Added README

ericl · 2017-10-12T04:51:30Z

python/ray/tune/trial.py

+        agent_cls = get_agent_class(self.alg)
+        cls = ray.remote(num_gpus=self.resources.get('gpu', 0))(agent_cls)
+        self.agent = cls.remote(
+            self.env_creator, self.config, self.local_dir,


ericl · 2017-10-12T04:51:57Z

python/ray/tune/trial.py

+    def train_remote(self):
+        """Returns Ray future for one iteration of training."""
+
+        assert self.status == Trial.RUNNING, self.status


The second status is printed out if the first expression is false?

ericl · 2017-10-12T04:52:40Z

python/ray/tune/trial.py

+        return self.agent.train.remote()
+
+    def should_stop(self, result):
+        """Whether the given result meets this trial's stopping criteria."""


Yeah, the user can just invert their objective for now

AmplabJenkins · 2017-10-12T05:32:35Z

Merged build finished. Test FAILed.

AmplabJenkins · 2017-10-12T05:32:36Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2101/
Test FAILed.

AmplabJenkins · 2017-10-12T05:37:24Z

Merged build finished. Test FAILed.

AmplabJenkins · 2017-10-12T05:37:24Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2102/
Test FAILed.

AmplabJenkins · 2017-10-12T05:54:40Z

Merged build finished. Test FAILed.

AmplabJenkins · 2017-10-12T05:54:41Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2103/
Test FAILed.

robertnishihara · 2017-10-12T07:09:12Z

I'm seeing some errors in Jenkins, not sure if this is the "main" error, but I see, e.g.,

    alg1 = cls("CartPole-v0", config)
  File "/opt/conda/lib/python2.7/site-packages/ray-0.2.1-py2.7-linux-x86_64.egg/ray/rllib/common.py", line 105, in __init__
    "all agent configs: {}".format(k, self.config.keys()))
Exception: Unknown agent config `episodes_per_batch`, all agent configs: ['exploration_fraction', 'hiddens', 'prioritized_replay_beta0', 'schedule_max_timesteps', 'num_workers', 'double_q', 'target_network_update_freq', 'prioritized_replay_eps', 'timesteps_per_iteration', 'sample_batch_size', 'buffer_size', 'model', 'prioritized_replay_alpha', 'dueling', 'grad_norm_clipping', 'num_cpu', 'print_freq', 'train_batch_size', 'lr', 'learning_starts', 'exploration_final_eps', 'gpu_offset', 'prioritized_replay_beta_iters', 'gamma', 'prioritized_replay']
Disconnecting client on fd 11

robertnishihara · 2017-10-12T07:12:53Z

test/trial_runner_test.py

@@ -0,0 +1,200 @@
+from __future__ import absolute_import


Looks like we aren't running this test anywhere. Should we run it in Travis (that would require pip installing gym in Travis)? or Jenkins?

We may want to go with Jenkins since we'll be able to run longer jobs (it currently lacks Python 3 testing, but we can build a docker container with Python 3 and potentially do both in Jenkins).

This test only takes 45s since we use mocked agents (and it shouldn't depend on gym).

How do I enable it in travis?

Basically need to add it somwhere around here

ray/.travis.yml

Line 122 in 3764f2f

- python -m pytest python/ray/rllib/test/test_catalog.py

AmplabJenkins · 2017-10-12T18:59:35Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-10-12T18:59:36Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2105/
Test PASSed.

AmplabJenkins · 2017-10-13T06:28:24Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-10-13T06:28:25Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2112/
Test PASSed.

AmplabJenkins · 2017-10-13T19:11:34Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-10-13T19:11:34Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2114/
Test PASSed.

ericl added 11 commits October 10, 2017 14:18

clean up train

26c54e7

update

4ba055a

update train script

98366ba

add tuned examples

9db66a8

add agent catalog

c65c379

add tune lib

dde75b5

update

c3fded2

fix

5c31f70

testS

c457d66

remove

bfd7c94

train docs

54b196f

ericl assigned pcmoritz and richardliaw Oct 11, 2017

richardliaw reviewed Oct 11, 2017

View reviewed changes

pcmoritz reviewed Oct 11, 2017

View reviewed changes

richardliaw reviewed Oct 12, 2017

View reviewed changes

comments

fbbd9f0

ericl force-pushed the ray-tune branch from 4b3459c to 4329457 Compare October 12, 2017 05:12

todo

81ad34f

ericl force-pushed the ray-tune branch from 4329457 to 81ad34f Compare October 12, 2017 05:13

ericl commented Oct 12, 2017

View reviewed changes

fix resource parsing

81dfd3e

robertnishihara reviewed Oct 12, 2017

View reviewed changes

fix cr test

42186b2

add test

865007f

try to fix travis test

4995ae2

pcmoritz approved these changes Oct 13, 2017

View reviewed changes

ericl merged commit 79ea205 into ray-project:master Oct 13, 2017

This was referenced Oct 14, 2017

[rllib] Results should be comparable with TensorBoard #919

Closed

[rllib] Add the ability to run arbitrary Python scripts with ray.tune #1132

Merged

[rllib] Initial work on integrating hyperparameter search tool #1107

[rllib] Initial work on integrating hyperparameter search tool #1107

Conversation

ericl commented Oct 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Oct 11, 2017

AmplabJenkins commented Oct 11, 2017

Choose a reason for hiding this comment

pcmoritz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Oct 12, 2017

AmplabJenkins commented Oct 12, 2017

AmplabJenkins commented Oct 12, 2017

AmplabJenkins commented Oct 12, 2017

AmplabJenkins commented Oct 12, 2017

AmplabJenkins commented Oct 12, 2017

robertnishihara commented Oct 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Oct 12, 2017

AmplabJenkins commented Oct 12, 2017

AmplabJenkins commented Oct 13, 2017

AmplabJenkins commented Oct 13, 2017

AmplabJenkins commented Oct 13, 2017

AmplabJenkins commented Oct 13, 2017

ericl commented Oct 11, 2017 •

edited

Loading