Add policies arg callback on_episode_step

[jsuarez5341]

Inconsistent callback signature: on_episode_step currently missing the "policies" arg. This prevents an important vector for smuggling data from the policy into the environment. A concrete example of this: Neural MMO environments render policy data as overlays on top of the env and need a way to get value function data etc to the render client.

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[jsuarez5341]

@sven1977 PR as requested

[sven1977]

Hey @jsuarez5341 , thank you! :)
Just added 2 things:

• Make the policy arg None by default. That should make all the testing errors go away.
• Use the policy arg to where the callback is actually done (in sampler.py).

[sven1977]

Thanks for the fix @jsuarez5341 ! :)

[sven1977]

Just waiting for tests to pass ...