How to define whether search (or train flag) is enabled?

[albertz]

How to define whether search is enabled? Just not do any special handling on this level at all and leave it to the current RETURNN behavior? Or make it more explicit?

From what I can see RETURNN handling should be enough. Assuming using this with automation changes in config can be done elsewhere I think.

Note that this is not just about what is enough to be able to define all networks. Or not sure how you mean it.

It's about being straight forward and clear. I.e. at no time, it should be unclear to the user when search is used.

We do not have to follow exactly the behavior of RETURNN. There are also multiple ways in RETURNN. We can restrict it to one clean way. We can also change it. Or introduce a simpler variant here.

I'm tending to make it explicit. But not sure.

PyTorch also has a similar concept for the train flag (as we do have as well in RETURNN). I.e. some PyTorch modules behave differently depending if they are in train or eval mode (e.g. Dropout). We have exactly the same in RETURNN. And search is a flag like train.

The difference is how these flags are set:

• In RETURNN, this is all globally, and for search flag, there are some additional (maybe unintuitive) ways to overwrite it. And the flags are implied automatically in RETURNN, depending e.g. on the task, and the user has not much control over it. It is quite hidden.

• In PyTorch, there are no implicit automatic implied global flags. Every module has its own flag, and it is set explicitly (and easily recursively for all sub modules). Every module has always the train flag set initially, and you can disable it explicitly. So to the user, it's always clear how the flags are set, because the user sets them, and no automatic behavior. The user explicitly writes model.train() or model.eval().

Maybe again, here in returnn-common, we can follow the PyTorch style a bit for this, and also copy it for the search flag? Not sure...

Originally posted by @albertz in #16 (comment)

[Atticus1806]

Maybe again, here in returnn-common, we can follow the PyTorch style a bit for this, and also copy it for the search flag? Not sure...

If I understand this correctly, each module has an internal flag which defines wether this (and only this module) is in train or search mode. I like this more than a global flag. This makes checking within the module easy to apply different versions of the network. In my Tacotron2 version right now I am passing a flag when calling the root module, but this is quite ugly and I would prefer a way that set actually part of returnn_common.

That being said maybe some questions that come to mind right now:

• Is it only a binary flag? RETURNN has more than search and train as flags, but do we need to differentiate between them here? Without deeper understanding on how modes like forward in RETURNN work I would tend to saying yes, but depending on internal handling maybe no.
• What would be functions the Module class would need to implement? get, set, set_recursively (to all Modules used in this), maybe also something like lock preventing external modification (via e.g. set_recursively)?
• Would this also give the chance to change the number of inputs for the forward function, since this might in some cases change when doing search (some information that is given as data during training but which the model has to generate itself during search? I am sure there are examples)

[albertz]

On PyTorch: there is a training: bool flag on each Module, which is True by default. There is the Module.train(mode=True) function which sets self.training = mode and successively also calls module.train(mode) for all child modules. There is the Module.eval() function which just calls self.train(False). As far as I know, the only PyTorch base modules which make use of this flag are Dropout and BatchNorm. But other third-party modules might make use of it as well. It is standard that you call model.eval() first when you do evaluation.

So, that's PyTorch.

RETURNN has more flags, currently: train_flag: Union[bool, tf.Tensor], eval_flag: bool, search_flag: bool.
We do not necessarily have train_flag == not eval_flag.
eval_flag means that RETURNN calculates the losses (does evaluation).
train_flag has influence on behavior of dropout and maybe other layers (just like in PyTorch).
train_flag implies that we do training, so it implies that we need to calculate the losses, so it implies eval_flag=True.
train_flag can be dynamic (tf.Tensor, a placeholder) (such that we have a single computation graph both for training and evaluation).
search_flag enables the search (ChoiceLayer, and the search choices logic, etc).
Those flags are per network, not per layer. But we can easily extend that in RETURNN if we want to.

The question is, what do we do here in returnn-common?

Also, I think there is some discussion on PyTorch side whether the training flag is really so useful. I think most user code anyway explicitly constructs the model either for training, evaluation, or whatever else, and just doesn't use this flag for this. Because as you say, it's often not just these two cases (binary) but other cases as well. And this is hard to generalize. Different applications, models, methods might have very different need, different kind of flags or options.

[Atticus1806]

So you would actually argue, that it would be either really complicated in usage (so not straight forward as we want returnn_common to be) or to simple in logic (not catching all possibilities) to use such a flag, which in both cases will most likely make the users default to writing some simpler logic for their use case? I can actually see that. But maybe it makes sense to give a simple straight forward logic for the base cases, and then leave more complex cases to the user?

[albertz]

I'm unsure.

We could adopt the training flag just as in the same way as in PyTorch.
(eval_flag is anyway not really needed separately, as we handle losses more explicitly here.)
Or not and just leave that logic to RETURNN.

And then not introduce other flags in the base Module but leave that as specific options to modules.
Or introduce a search flag with similar logic as the training flag. Although maybe this recursive setting the flag to all child modules is not so useful or necessary for this?

Remember, our goal is that it is straightforward for the user, with priority on understanding and reading existing code, but writing new code should be straightforward as well.

It's hard to say which way would be the most straightforward.

We maybe should do a bit of research and look for common usage patterns in other frameworks. How do they deal with the difference between training, search, forwarding, maybe other cases? E.g. look at Fairseq, ESPNet, others.

[albertz]

Note that the RETURNN train_flag is also used to differentiate between the standard training and cross validation. This is an important distinction. Currently RETURNN constructs one single network (computation graph) for both training including cross validation.

If we make everything completely independent from the RETURNN train_flag here, then we need to extend RETURNN to be able to define a separate network for cross validation. Maybe that's anyway a good idea, to have that explicit?

If we want to keep the RETURNN train_flag logic, we can also access the train_flag dynamically (via a RETURNN layer) and then use the standard conditional logic (#24) to define custom behavior depending on this flag (e.g. for batch norm or dropout).