[RLlib] Mutiagent learning: can't combine replay lockstep and multiple agents controlled by the same policy.

[raphaelavalos]

Hello,

While implementing MADDPG for PyTorch I noticed that it is not possible to combine the lockstep replay mode configuration and having multiple agents controlled by the same policy. This is due to the implementation of MultiAgentBatch as a dict of PolicyID -> SampleBatch. In the contrib TensorFlow implementation this issue was circumvented using parameter sharing. Another solution which is framework agnostic would be to group the agents.

I think that this combination should be supported without needing to group the agents. However, I suspect that it would require significant code change. Maybe another solution exists?

Also, the documentation should be updated alongside with the code that checks that the config is valid.

Thanks
(Amazing work btw)

[ericl]

Hmm, I think this should be ok. The batch will indeed be keyed by policy id, but you can look into the "agent_id" field of the batch to see which experiences correspond with which original agent.

[raphaelavalos]

Okk I will look into that next week and let you know. Thx

[raphaelavalos]

I will update the docs tomorrow and make a PR.