How to define the API for parameter initialization, regularization (L2, weight dropout, etc), maybe updater opts per-param

[albertz]

It is maybe not such a nice idea if every new Module will have this option and then explicitly passes it on to all submodules. Some modules might also not implement this. And maybe there are other options which are not handled yet, e.g. param noise, etc.

How should this be handled? Mostly such arguments which every layer can potentially accept. Mostly arguments about certain behavior on parameters (variables).

In TF, the natural way would be to use tf.variable_scope context scope and there have some custom getter or other custom logic.

So maybe also some context manager here?

[albertz]

Also Lstm with rec_weight_dropout. That should be removed, and we should have a generic weight_dropout.
Although such generic weight dropout still needs to be implemented on RETURNN side (rwth-i6/returnn#735).

[albertz]

The same also for parameter initialization options. Like forward_weights_init and bias_init.
Maybe we should also extend RETURNN in some way to make this more generic. The LayerBase.var_creation_scope could be extended by some generic mechanism.

[albertz]

LinearLayer also has filter_grad. This is applied after the matrix multiplication, before the optional activation function.
This should be more generic as well. Basically this could be applied after every layer.
There could be a RETURNN layer for this, like FilterGradLayer, which would be somewhat similar to ScaledGradientLayer.
Or this could be a generic option for LayerBase. And then applied in LayerBase.post_init.

Then this becomes also relevant here.

[albertz]

LayerBase also has trainable. Should we also handle that here?

[albertz]

LayerBase also has updater_opts. Should we also handle that here?

[albertz]

LayerBase also has param_variational_noise, which should be handled here. However, this applies to all params of a layer. Maybe we want to have more fine-grained support for each param?

[albertz]

There are some other potential scope-based or global options (all potential LayerBase options) like param_device.
PyTorch also has this, like device or dtype (e.g. to use bfloat16), like:

with torch.cuda.device(1):
  ...

or device and dtype are also arguments to most modules in torch.nn.