Losses to implement and losses naming conventions

[albertz]

Some modules we should implement:

• CrossEntropy or CE. Should this cover both dense and sparse targets, or do we want a separate module for the sparse case, like SparseCrossEntropy or so? Should this potentially also allow for logits? log-probs?
• KL or KullbackLeiblerDivergence
• BinaryCrossEntropy or BCE
• L2Dist (absolute or mean?) Or MSE or MeanSquarredError? (The mean reduction is over the feature axis. Not over time or batch.)
• L1Dist (absolute or mean?) Or MeanL1Dist?
• Ctc or CTC or CtcLogProb
• CosineSimilarity

I don't like the naming of the PyTorch losses too much here.
They have the postfix Loss on all of them, although these modules are generic and not necessarily just for loss computation (although that's probably their most common usage).
Also CrossEntropyLoss is actually log-softmax + CE together. So very much like the TF tf.nn.sparse_softmax_cross_entropy_with_logits.
And there is a separate NLLLoss. Which is just like CrossEntropyLoss but it doesn't take logits but log-prob instead. I find this naming to be confusing.

Also the question is how we should handle things like label smoothing. On RETURNN side (and also in TF), it is just an option to the CE-loss. On PyTorch side, it is not implemented yet as part of the official PyTorch API. Some background here. It was only very recently added (pytorch/pytorch#7455, pytorch/pytorch#63122). This also adds it as an option label_smoothing to CrossEntropyLoss. An alternative would be that the user makes this more explicit, like:

target_prob_smooth = smooth_one_hot(targets, label_prob=0.9)
loss = cross_entropy(target_prob_smooth, out_prob)

Although label smoothing has become very common, so maybe it makes sense to have this also just as an option.

Note also that the loss accumulation over the dataset and handling of calculating the correct average (mean) is handled by RETURNN. All such losses would just yield a vector of shape [B] or [B,T].

[Atticus1806]

I don't like the naming of the PyTorch losses too much here.

I think this changed from the last time you looked? Because looking into https://pytorch.org/docs/stable/nn.html#loss-functions they do have Loss as postfix.

I assume we want to put them similar to PyTorch into a sperate file like loss.py.

What would you think of as tests for the losses? Because I think this issue is something we should be careful about, when implementing by hand. Maybe run a small RETURNN with the RETURNN and the rc (Module) loss and compare the results? Does that catch all cases though?

[albertz]

I don't like the naming of the PyTorch losses too much here.

I think this changed from the last time you looked? Because looking into https://pytorch.org/docs/stable/nn.html#loss-functions they do have Loss as postfix.

But that is one thing which I meant. This is what I don't like. CE is not just a loss. It is just CE. You can use it as a loss. But so can you also use anything else as a loss. And you can also use CE for other purpose, not just as a loss.

I made this comment to say that we maybe should not exactly follow the PyTorch naming scheme in this case. Or at least we should think about it.

I assume we want to put them similar to PyTorch into a sperate file like loss.py.

Yea sure, but this would just be our internal structuring, so this doesn't really matter too much. The user would simply use rc.nn.CE or whatever.

What would you think of as tests for the losses? Because I think this issue is something we should be careful about, when implementing by hand. Maybe run a small RETURNN with the RETURNN and the rc (Modul) loss and compare the results? Does that catch all cases though?

I don't quite understand. What exactly do you want to test? That e.g. the CE loss here and the RETURNN CE loss yield the same result? You can just write a test for that. Or for whatever else you want to test.

[Atticus1806]

Ohh I missread that, my bad.
Looking into it in PyTorch it seems inconsistent to me too. In their Modules they just call the functions from torch.nn.functional and in these functions e.g. CE does not have the loss postfix, but e.g. CTC still has the postfix. Maybe we can wrap in a similar way but then consistently have all the functions not named like losses?

What exactly do you want to test?

I think we should think about good tests for the losses since they are crucial part of optimization. My suggestion assumed we see the RETURNN losses as correct and test based on that for similar results so we can infer the correctness of the Modules. I just wanted to say that we should think about it already when writing the Modules and not afterwards. But maybe this is something that is not as hard as it seems to me right now?

[albertz]

Maybe we can wrap in a similar way but then consistently have all the functions not named like losses?

What do you mean by wrapping? We would not really wrap anything here.
The naming is completely up to us. So you also prefer the variant without Loss postfix?

[albertz]

I think we should think about good tests for the losses since they are crucial part of optimization. ... But maybe this is something that is not as hard as it seems to me right now?

Why should it be hard to write a test case? I don't really see any problem.

[Atticus1806]

What do you mean by wrapping?

class L1Loss(_Loss):
    __constants__ = ['reduction']

    def __init__(self, size_average=None, reduce=None, reduction: str = 'mean') -> None:
        super(L1Loss, self).__init__(size_average, reduce, reduction)
    def forward(self, input: Tensor, target: Tensor) -> Tensor:
        return F.l1_loss(input, target, reduction=self.reduction)

This is the Loss class which then refers to the loss in torch.nn.functional which then refers to a C implementation.

So you also prefer the variant without Loss postfix?

For the implementation of the function yes, but we should have a Module that just calls this function with the Loss Postfix because I think it is straight forward when just wanting to use it as a loss, to reflect that in the name (which ofc. is easy to do).

Why should it be hard to write a test case? I don't really see any problem.

I did not put that much thought into it yet, I just could not come up with a quick idea on how to test, but if you say its not hard then its fine.

[albertz]

What do you mean by wrapping?

class L1Loss(_Loss):...

This is the Loss class which then refers to the loss in torch.nn.functional which then refers to a C implementation.

But this is PyTorch. You wrote:

we can wrap in a similar way

What should we wrap? Why do we need to wrap anything? I don't think we need to wrap anything.

[Atticus1806]

What should we wrap? Why do we need to wrap anything? I don't think we need to wrap anything.

My suggestion was to implement the loss function without loss postfix as a standard function (like you suggested) and then have a class with the loss name similar to PyTorch, because while I agree with you that the functions can be used without their Loss intention, I feel like for a lot of users it would be more straight forward to actually have Loss in the name. Maybe this can already do e.g. mark_as_loss internally or similar.

[albertz]

For the implementation of the function yes, but we should have a Module that just calls this function with the Loss Postfix because I think it is straight forward when just wanting to use it as a loss, to reflect that in the name.

But I don't understand that reasoning.

• You want it both as a function and as a module? Why? Do we really want that for everything now? Or just for losses? (Related: Functional layer API, conventions #21. You are right, we currently do have it for Dropout & dropout and some others.)
• You want that functions and modules are inconsistent to each other? The module name should have the Loss prefix, but not the function? Why?

[Atticus1806]

Oh now I think I see your Problem. No I don't necessarily want one Module and one Function (that was bad wording I guess), in general this could also be 2x Module or 2x Function, but I would argue for having that "Non-Loss" Version wrapped in a "Loss" Version which handles the potential Loss settings.