[rllib] Added evaluation script to RLLib #1295

pschafhalter · 2017-12-06T01:30:04Z

What do these changes do?

Added evaluation script to RLLib
Added test for evaluation script

Related issue number

[RLlib] Write script that can visualize policy from checkpoint #1280

AmplabJenkins · 2017-12-06T01:56:19Z

Merged build finished. Test FAILed.

AmplabJenkins · 2017-12-06T01:56:20Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2643/
Test FAILed.

AmplabJenkins · 2017-12-06T21:05:55Z

Merged build finished. Test FAILed.

AmplabJenkins · 2017-12-06T21:05:56Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2651/
Test FAILed.

ericl · 2017-12-06T21:48:03Z

test/jenkins_tests/run_multi_node_tests.sh

@@ -139,6 +139,9 @@ docker run --rm --shm-size=10G --memory=10G $DOCKER_SHA \
    --stop '{"training_iteration": 2}' \
    --config '{"num_workers": 2, "use_lstm": false, "use_pytorch": true, "model": {"grayscale": true, "zero_mean": false, "dim": 80, "channel_major": true}}'

+docker run --rm --shm-size=10G --memory=10G $DOCKER_SHA \
+  sh /ray/test/jenkins_tests/multi_node_tests/test_rllib_eval.sh


ericl · 2017-12-06T21:51:04Z

python/ray/rllib/eval.py

+required_named.add_argument(
+    "--env", type=str, required=True, help="The gym environment to use.")
+parser.add_argument(
+    "--hide", default=False, action="store_const", const=True,


--no-render

ericl · 2017-12-06T21:51:14Z

python/ray/rllib/eval.py

+
+EXAMPLE_USAGE = """
+example usage:
+    ./train.py /tmp/ray/checkpoint_dir/checkpoint-0 --run DQN --env CartPole-v0


This should say ./eval.py right?

ericl · 2017-12-06T21:51:44Z

python/ray/rllib/eval.py

+import gym
+import ray
+
+from agent import get_agent_class


from ray.tune.agent import

ericl · 2017-12-06T21:51:51Z

python/ray/rllib/eval.py

@@ -0,0 +1,52 @@
+#!/usr/bin/env python
+


Could you do the import future stuff from train.py as well here?

Should be fixed in the latest commit

ericl · 2017-12-06T21:52:09Z

python/ray/rllib/eval.py

+        "given a checkpoint.", epilog=EXAMPLE_USAGE)
+
+parser.add_argument("checkpoint", type=str,
+                    help="Checkpoint from which to evaluate.")


should make the indentation consistent

ericl · 2017-12-06T21:52:33Z

python/ray/rllib/eval.py

+        "user-defined trainable function or class registered in the "
+        "tune registry.")
+required_named.add_argument(
+    "--env", type=str, required=True, help="The gym environment to use.")


Do we also need --config?

ericl · 2017-12-06T21:53:28Z

python/ray/rllib/eval.py

+    env = gym.make(args.env)
+    state = env.reset()
+    done = False
+    while not done:


Consider doing it in an infinite loop.

richardliaw · 2017-12-07T09:39:20Z

python/ray/rllib/eval.py

+
+    cls = get_agent_class(args.run)
+    agent = cls(env=args.env)
+    agent._restore(args.checkpoint)


agent.restore is the proper way

AmplabJenkins · 2017-12-07T23:55:27Z

Merged build finished. Test FAILed.

AmplabJenkins · 2017-12-07T23:55:28Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2672/
Test FAILed.

AmplabJenkins · 2017-12-08T00:37:23Z

Merged build finished. Test FAILed.

AmplabJenkins · 2017-12-08T00:37:24Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2673/
Test FAILed.

pcmoritz · 2017-12-08T01:07:57Z

@pschafhalter Can you rebase this on the latest master so we can see if it passes the Jenkins test you added?

pschafhalter · 2017-12-08T01:25:28Z

@pcmoritz it should be rebased now

AmplabJenkins · 2017-12-08T02:06:22Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-08T02:06:23Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2676/
Test PASSed.

AmplabJenkins · 2017-12-08T02:12:18Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-08T02:12:18Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2677/
Test PASSed.

pcmoritz

@pschafhalter LGTM, now that #1294 is fixed, can you re-introduce the loop in the shell script? All checkpoints should now start with index 1.

pschafhalter · 2017-12-08T05:38:05Z

@pcmoritz are checkpoints for A3C fixed as well?

pcmoritz · 2017-12-08T06:13:08Z

@pschafhalter No, they are not I think. Do you want to create a github issue about it?

pschafhalter · 2017-12-08T06:22:31Z

@pcmoritz sure, I'll report it

ericl

LGTM. Do you also want to update the docs (rllib.rst)?

pschafhalter · 2017-12-08T08:04:07Z

@ericl yes, I'll update the docs as well. Should I create a new section or document the script under Getting Started?

ericl · 2017-12-08T17:12:38Z

Let's integrate it into getting started

…

On Fri, Dec 8, 2017, 12:04 AM Peter Schafhalter ***@***.***> wrote: @ericl <https://github.com/ericl> yes, I'll update the docs as well. Should I create a new section or document the script under Getting Started? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1295 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAA6SnwPeKRfJAmKriFy3MtPvXOJrPhVks5s-O15gaJpZM4Q3Opy> .

AmplabJenkins · 2017-12-09T22:44:42Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-09T22:44:43Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2698/
Test PASSed.

pschafhalter · 2017-12-09T22:53:00Z

@ericl I added some documentation. Let me know if there's anything I should change.

ericl · 2017-12-10T19:22:33Z

doc/source/rllib.rst

@@ -68,6 +68,35 @@ The most important options are for choosing the environment
 with ``--env`` (any OpenAI gym environment including ones registered by the user
 can be used) and for choosing the algorithm with ``--run``
 (available options are ``PPO``, ``A3C``, ``ES`` and ``DQN``).
+In order to save checkpoints from which to restore training 


Make this a subsection ("Evaluating trained agents") and move to below the specifying params section below?

ericl

Lgtm, one suggestion on organization since the added text is quite long.

AmplabJenkins · 2017-12-11T04:31:43Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-11T04:31:43Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2708/
Test PASSed.

pcmoritz requested a review from richardliaw December 6, 2017 01:36

ericl reviewed Dec 6, 2017

View reviewed changes

richardliaw changed the title ~~Added evaluation script to RLLib~~ [rllib] Added evaluation script to RLLib Dec 7, 2017

richardliaw reviewed Dec 7, 2017

View reviewed changes

pschafhalter force-pushed the rll-eval branch from 2906b21 to a7ed5e2 Compare December 8, 2017 01:24

pschafhalter added 9 commits December 7, 2017 17:26

Added eval script

c25094b

Added tests for eval script

ec16345

Fixed paths

d0e1de3

Add eval test to run tests script

0d2012d

Fix linting

ebd9ef3

Add future imports

938fc9f

Address PR comments

35ddce2

Fix bug and linting

a7ed5e2

Fix indent in running test

d21d236

pcmoritz self-requested a review December 8, 2017 04:32

pcmoritz reviewed Dec 8, 2017

View reviewed changes

ericl reviewed Dec 8, 2017

View reviewed changes

Updated test script to match fixed checkpoints (ray-project#1294)

b915195

Add documentation for eval.py

3e4e1b8

ericl reviewed Dec 10, 2017

View reviewed changes

ericl approved these changes Dec 10, 2017

View reviewed changes

Created subsection for eval.py

76664e5

richardliaw merged commit 20d6b74 into ray-project:master Dec 11, 2017

pschafhalter deleted the rll-eval branch April 20, 2018 20:50

[rllib] Added evaluation script to RLLib #1295

[rllib] Added evaluation script to RLLib #1295

Conversation

pschafhalter commented Dec 6, 2017

What do these changes do?

Related issue number

AmplabJenkins commented Dec 6, 2017

AmplabJenkins commented Dec 6, 2017

AmplabJenkins commented Dec 6, 2017

AmplabJenkins commented Dec 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Dec 7, 2017

AmplabJenkins commented Dec 7, 2017

AmplabJenkins commented Dec 8, 2017

AmplabJenkins commented Dec 8, 2017

pcmoritz commented Dec 8, 2017

pschafhalter commented Dec 8, 2017

AmplabJenkins commented Dec 8, 2017

AmplabJenkins commented Dec 8, 2017

AmplabJenkins commented Dec 8, 2017

AmplabJenkins commented Dec 8, 2017

pcmoritz left a comment

Choose a reason for hiding this comment

pschafhalter commented Dec 8, 2017 • edited Loading

pcmoritz commented Dec 8, 2017

pschafhalter commented Dec 8, 2017

ericl left a comment

Choose a reason for hiding this comment

pschafhalter commented Dec 8, 2017

ericl commented Dec 8, 2017 via email

AmplabJenkins commented Dec 9, 2017

AmplabJenkins commented Dec 9, 2017

pschafhalter commented Dec 9, 2017

Choose a reason for hiding this comment

ericl left a comment

Choose a reason for hiding this comment

AmplabJenkins commented Dec 11, 2017

AmplabJenkins commented Dec 11, 2017

pschafhalter commented Dec 8, 2017 •

edited

Loading