Improve the PPOLearner to support continuous action space

[findmyway]

Necessary changes:

1. Add a field of dist into the PPOLearner (just like @norci did in VPG )

2. Following method needs to be extended to recognize environments with continuous action space. Currently the PPOLearner is assumed to return a (batch of ) logits. I'd suggest to rename the PPOLearner into PPOPolicy and return an action directly.
   

   ReinforcementLearningZoo.jl/src/algorithms/policy_gradient/ppo.jl

   Line 77 in 34aea90

   function (learner::PPOLearner)(env::MultiThreadEnv)

3. GausianNetwork is also needed.

4. Calculating entropy loss in update! is hard coded. Better to split it into a function to support continuous distribution (or reuse the one in StatsBase or Distributions. But use them with caution! I had some problems using them with Zygote before)

[norci]

I think tianshou's implementation is a good reference:

https://github.com/thu-ml/tianshou/blob/master/test/continuous/test_ppo.py
https://github.com/thu-ml/tianshou/blob/master/tianshou/utils/net/continuous.py

[norci]

I found a few papers about continuous control. they did not use Gaussian policy.

• http://proceedings.mlr.press/v70/chou17a.html
• https://deepai.org/publication/continuous-control-with-deep-reinforcement-learning
• https://deepai.org/publication/learning-continuous-control-policies-by-stochastic-value-gradients
• https://arxiv.org/abs/1802.07564

I'm still trying to understand these papers.

[stasti]

i think to have Gaussian policy is useful at least as the baseline before jumping to more advanced methods. and it is still used in many works.