Allow anonymous parameters?

[albertz]

Currently all parameters must have names, i.e. they mast be accessible from the root module via an attribute chain (like root_mod.sub_mod.param). This has many advantages, such as:

• unique and straightforward name in the model checkpoint
• easy iteration over all params via root_mod.parameters() or root_mod.named_parameters()
• easy param sharing (within the same module, or also across modules)

In some cases, anonymous parameters (not accessible via attrib chain from the root module) can be useful though:

• It is required to allow a function like nn.random_normal because this has internal state (i.e. a param holding the state). Otherwise you must always explicitly create random number generators as submodules via nn.Random.

• Some people sometimes prefer to write modules in a more functional-style way, where you define functions like self_attention_block, where nn.Linear etc are only locally created.

So, what should we do? Be strict about this and never allow this?
Or if we allow it, some followup questions:

• Should these params be stored in the model checkpoint? How do we define the name?
• Should it be possible to iterate over the anonymous params? Via a separate function, nn.global_params or so?

Maybe anonymous functions could automatically be added to the enclosing module (determined via scope) by some automatic naming scheme. Then root_module.parameters() would also cover them, and they would be stored in the checkpoint as well.

[albertz]

@Zettelkasten @Atticus1806 @JackTemaki opinions?

[JackTemaki]

I think it is fine to not allow anonymous submodules/parametrized layers. I think the benefits do not outweight the needed addition in code for the automatic handling.

[albertz]

Just to emphasize again: Not allowing this means, we cannot have any such functions:

• nn.dropout
• nn.self_attention or nn.cross_attention with attention_dropout argument
• nn.random_uniform etc
• nn.spec_augment

Basically anything which uses some random op at some point must be a nn.Module.

[Atticus1806]

I am not sure about the implications, but I have to say that the way e.g. dropout is used right now is really nice.

Currently all parameters must have names

I am not sure if I understand. So from what I understand explicit naming forbids functions like nn.dropout, but this is possible right now. Or is this naming more a convention right now and we still allow anonymous params?

[albertz]

I am not sure if I understand. So from what I understand explicit naming forbids functions like nn.dropout, but this is possible right now. Or is this naming more a convention right now and we still allow anonymous params?

The reason why it is possible right now is because it wraps the RETURNN DropoutLayer, so the random state is never exposed in any way (and thus also not handled perfectly correct, just like in all our old models), so there is no param.

If we would implement it in a correct way (maybe we should anyway to have the random state handled correctly), we would have this problem.

Despite, I think currently anonymous parameters are actually allowed. When you just run nn.random_normal(()), you will not get an error. Only you will not be able to access it from outside. And most/all tests explicitly check the params in check_params() and would raise an error in such cases.

[albertz]

Btw, regarding dropout: I think we anyway should also reimplement dropout, to stay consistent and have the same behavior for randomness as for other things. So, if we decide here to not allow anonymous params, it means we cannot have nn.dropout anymore but we would have a nn.Dropout module.