DDPG example uses BatchNormalization incorrectly

[lubiluk]

Correct me if I'm wrong, but according to the documentation of tf.keras.layers.BatchNormalization it functions differently depending on whether or not training is set to True.
In the example code ddpg_pendulum.py this mode is never altered. Effectively, I think, this means that normalization has no effect.

[fchollet]

Good call, we should add training=True arguments to model calls that happen under the gradient tape. @amifunny could you please fix this?

[amifunny]

@fchollet Isn't the training=true default argument when passing input to model? Which I think is ok, not to define explicitly as is the case in all the TensorFlow tutorials.

@lubiluk In our example, we are not testing our model at all, we are just examining if our reward is increasing and assuring that the DDPG Algorithm works. If we were to export this model to test, yes we would use training=False so that we use a previous running average of dimension values and make testing faster.

Effectively, I think, this means that normalization has no effect.

If you would look at weights being trained, Normalization layer weights are being trained.

[fchollet]

@fchollet Isn't the training=true default argument when passing input to model? Which I think is ok, not to define explicitly as is the case in all the TensorFlow tutorials.

No it defaults to False. It is accurate that in the code as written currently, BN does nothing.

[fchollet]

If you would look at weights being trained, Normalization layer weights are being trained.

Its trainable weights only. So to be precise, you're not doing BN, you're just doing featurewise scaling and centering with learned coefficients.

[amifunny]

Its trainable weights only. So to be precise, you're not doing BN, you're just doing featurewise scaling and centering with learned coefficients.

Yes, If the default is false then the layer was just doing that. I have submitted a PR #246 for the correction.

[lubiluk]

I think in policy() training should be False in order to use normalization here. IMO, acting in the environment is not part of the training but rather an evaluation. If we use training=True everywhere then we also loose BN benefits.

[fchollet]

Solved by removing BN. Activating it prevents training altogether.