[rllib] Upgrade to OpenAI Gym 0.10.3

[ghost]

What do these changes do?

Upgrade rllib to to be compatible with OpenAI Gym 0.10.3.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3940/
Test FAILed.

[richardliaw]

Hi @butchcom, what version of Gym are you using? It seems that this patch isn't passing our tests.

[ghost]

I'm using version 0.9.7 (the newest), which was released just recently. In this release, the underscored method names were removed (see Changelog).

[richardliaw]

I see - perhaps we should consider updating the Gym version in tests. @ericl?

[ericl]

Upgrading makes sense. I believe we pinned to a previous version since the release was broken but presumably that's fixed.

[ghost]

@ericl you may be referring to #1486, but if I understand the testing setup correctly, #1460 already introduced the new Gym version afterwards. Interestingly, the old Gym version is still used in https://github.com/ray-project/ray/blob/master/docker/examples/Dockerfile.

Also, https://github.com/ray-project/ray/blob/master/python/ray/rllib/dqn/common/wrappers.py will probably also need to be updated.

[richardliaw]

Ah Travis tests don't actually touch the RLlib code, but Jenkins does. So #1460 didn't actually do anything wrt Gym.

That being said, if you're up for it, you can remove the pin in the docker file and push again to see if the tests will pass.

[ghost]

Ok, will try

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4024/
Test FAILed.

[richardliaw]

Looks like ES is the main source of failure:

File "/ray/python/ray/worker.py", line 762, in _process_task
    *arguments)
  File "/ray/python/ray/actor.py", line 209, in actor_method_executor
    method_returns = method(actor, *args)
  File "/ray/python/ray/rllib/es/es.py", line 116, in do_rollouts
    rewards_pos, lengths_pos = self.rollout(timestep_limit)
  File "/ray/python/ray/rllib/es/es.py", line 85, in rollout
    add_noise=add_noise)
  File "/ray/python/ray/rllib/es/policies.py", line 23, in rollout
    env_timestep_limit = env.spec.tags.get("wrapper_config.TimeLimit"
AttributeError: 'NoneType' object has no attribute 'tags'

[ghost]

It seems that way... Maybe https://github.com/openai/gym/blob/master/gym/envs/registration.py#L52 is the cause. Will investigate this further tomorrow.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4035/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4036/
Test FAILed.

[ghost]

The basic tests (in https://github.com/ray-project/ray/blob/master/python/ray/rllib/test/test_supported_spaces.py) seem to work now. The examples don't work with the new Gym version, though. I'll update these as well, along with https://github.com/ray-project/ray/blob/master/python/ray/rllib/dqn/common/wrappers.py.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4037/
Test PASSed.

[ghost]

Great, the upgrade seems to work. Now the only remaining warnings are of the kind: gym.spaces.Box autodetected dtype as <type 'numpy.float32'>. Please provide explicit dtype.. Should I fix these warnings as well?

[richardliaw]

hm where is the warning being raised?

[ghost]

The new Gym version added a dtype parameter to the Space constructor. For the Box space, dtype should be defined explicitly. If it isn't defined, float32 (or uint8 for visual state spaces) is used.

The rllib algorithms will work without explicitly defining dtype. But it would be easy to define the dtypes, in order to make these warnings disappear.

[richardliaw]

I see - so the fix would be to use uint8 dtypes for the visual wrappers and float32 for others? If so, that seems reasonable.

[ghost]

That's right. Then I'll implement it prior to the merge

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4045/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4134/
Test PASSed.

[richardliaw]

Tried it out, this is fine.

[richardliaw]

works on Linux + Mac py3.6 and tests pass, so I'll merge this.

@butchcom, thanks a bunch for contributing this!