WIP: [Impala] Scalable Distributed Deep-RL with ImportanceWeighted Actor-Learner Architectures

[joneswong]

This is a work-in-progress PR for implementing Impala, a scalable distributed reinforcement learning algorithm.
The impala optimizer has been implemented.
To do

• Impala agent (v-trace)
• support usage of LSTM and deep residual network
• multi-gpu support (allreduce)

and after all the above items have been done, we must carefully test impala via reproducing experimental results reported by original paper.

Related issue number

1924

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5654/
Test PASSed.

[robertnishihara]

Let's add

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

to the top of the file

[robertnishihara]

Looks like we forgot before, but let's also add

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

to the top of this file

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5779/
Test FAILed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5850/
Test PASSed.

[ericl]

FYI, #2299

will add efficient batched LSTM support similar to that described in the IMPALA paper.

[robertnishihara]

@ericl @joneswong is this PR still relevant or should it be closed?