[GSoC-Week3] Batch environments (cont.)

[Sid-Bhatia-0]

cc @findmyway @jonathan-laurent

Last week:

Worked on batch implementation of environments at #146.

This week:

This week, I will make necessary performance improvements and suggested changes to the single-threaded version of the batch environment SingleRoomUndirectedBatch. It is only after we have a performant single-threaded version of the batch environment that we should worry about CPU multi-threading or GPU.

The single-threaded batch environment with num_envs = 1 needs to be compared with a non-batch single environment version. If the performances are roughly equivalent, this means that we can translate all non-batch versions to batch versions without significantly sacrificing performance. In this case, we need only maintain one version (batch version) for each environment.

Once this is done, we can think about incorporating CPU multi-threading. GPU implementation will have to wait, because it is a whole different ball game. There is a bunch of work #146 #150 #149 #147 #148 already spawned up that needs to be prioritized and executed. I would like to tie these loose ends before venturing too deep into the high-performance side.

[Sid-Bhatia-0]

We now have some data to compare at #146 (comment)

We will need to decide if we should have both non-batch and batch versions maintained separately for each environment because of the performance differences? From the benchmarks, it seems like batch with num_envs = 1 is worse than non-batch. But the absolute values of either of these numbers are pretty small, raising the question of whether their differences would really matter in an end-to-end scenario or not.

Also, before we move on to create a multi-threaded or GPU implementation, It might be a good idea to create an end-to-end benchmark that concretely demonstrates the significant performance gains we are expecting from the struct of arrays representation of batch environments.

Simulating grid-world environments is very cheap. In order to really leverage batch environments, we need something that can handle such throughput and still scale roughly linearly with the number of environments. @findmyway you suggested that evolutionary strategies could make the cut, and after reading https://openai.com/blog/evolution-strategies/, I agree with you (even though in our case we are not presently dealing with distributed machines). Backpropagation is relatively more expensive than plain forward propagation, so deep learning might not be able to handle the throughput from the super cheap environment simulations that we can provide.