Log loss metric can be Infinity or NaN

[rogancarr]

For binary classification (and perhaps multiclass classification) the log loss can be infinite. The log loss reduction can also be negative infinity, as it is a shifting and rescaling of the log loss.

Similarly, the log loss can be a NaN. This is specifically guarded against in the code, but does seems like a bug too.

The culprit for both cases lies in the initial calculations in the ProcessRow() method of the Aggregator for the BinaryClassifierEvaluator.

Double logloss;
if (!Single.IsNaN(prob))
{
    if (_label > 0)
    {
        // REVIEW: Should we bring back the option to use ln instead of log2?
        logloss = -Math.Log(prob, 2);
    }
    else
        logloss = -Math.Log(1.0 - prob, 2);
}
else
    logloss = Double.NaN;

I propose that to guard against infinities we add an epsilon before taking the log.

To guard against NaNs, we will need to fix the probability calculations (e.g. in the calibrator(s)).

[TomFinley]

If I have an example in binary classification, and it has predicted probability 0 of being true, but actual label true, the log loss was in fact infinity. How do we reconcile that? When in the past we've used epsilons in this calculation, we often run into trouble because the choice of which epsilon becomes what amounts to a fairly arbitrary choice of what is the "maximum" log-loss per example. So there is a clear "right" answer (do the calculation that the user asked for, if they ask for log-loss, they get log-loss), but if they want 'clamped' log-loss maybe we can have an additional parameter for epsilon as sklearn does. I might stop shy of making the default value of that non-zero, though, since again that is an arbitrary choice. I don't insist on it though. Just so long as the correct calculation is possible.

NaN might be a bug or not. If someone feeds in a missing label, or the prediction step predicts nan probabilities, it's unclear to me that the output should be NaN or not. So it would depend on under what circumstances it is providing these nans, which is not yet clear from the description.

[rogancarr]

Hi @TomFinley! These are good points.

For NaN, I get what you're saying here. I'm more ambivalent about returning a NaN, but I do think it would be good to look into. Especially because this code does in fact check for NaN labels and such.

For the infinities, I am just surprised that anybody is predicting a perfect 1 or 0, given that these numbers are usually produced by a sigmoid function, and a sigmoid approaches +-infinity as a limit. This feels to me like a rounding error that bumps into infinity by accident.

The big problem is that this totally jams up the accumulation step. I can add up a lot of small numbers to counter a very large number, but I can't every get away from infinity. Similarly, once a NaN gets into the pipeline, then the whole calculation becomes a NaN.