AutoML.Net: CrossValSummaryRunner can't handle all-infinity metrics value.

[LittleLittleCloud]

CrossValSummaryRunner picks up the best model and score from cross validation in the following logic:

• get results from all validation runs
• if all run succeed, get the index of model with best score as return model, if all run's score is NaN or Infinity, uses the first model
• if all run succeed, get the score which is closest to the average score as return training score, if average score is NaN, use the first score (BUG)

The possible place that causes the bug is sometimes the average score can be Infinity, in which case the following function will return -1 and causes an index out of error exception

machinelearning/src/Microsoft.ML.AutoML/Experiment/Runners/CrossValSummaryRunner.cs

Line 84 in 37af3f9

var indexClosestToAvg = GetIndexClosestToAverage(trainResults.Select(r => r.score), avgScore);

Related issue

(The label column for this user's dataset is all empty, which is the root cause for this error)

• Model Builder throws index out of range error in regression when label column or feature column is all null machinelearning-modelbuilder#929

[antoniovs1029]

Hi, @LittleLittleCloud

So I'm not sure what's the expected behavior we should have. Do you want us to throw a more understandable exception message? Or is there actually something you think needs to be fixed?

The label column for this user's dataset is all empty, which is the root cause for this error

In this case, it makes complete sense that AutoML.NET throws an error.

The possible place that causes the bug is sometimes the average score can be Infinity, in which case the following function will return -1 and causes an index out of error exception

Is it possible to get an average score of infinity in a case different from having an empty label column?

I do see that we return index "0" if the average was NaN:

machinelearning/src/Microsoft.ML.AutoML/Experiment/Runners/CrossValSummaryRunner.cs

Lines 162 to 165 in 37af3f9

	// Average will be NaN iff all values are NaN.
	// Return the first index in this case.
	if (double.IsNaN(average))
	return 0;

I wouldn't know why that's the case, though. Do you think we should return "0" also if the average is Infinity?

[LittleLittleCloud]

Hi @antoniovs1029

For start, yes, I think we should return "0" if average is positive Infinity, negative infinity or NaN

In this case, it makes complete sense that AutoML.NET throws an error.

I think it's better not throw an error if the error is not from AutoML.Net. In this case, column all empty is not a good reason for throwing an error because from AutoML.Net side, train can still continue, it's just all the result will be NaN or infinity and doesn't have real meaning.

Is it possible to get an average score of infinity in a case different from having an empty label column

another example is empty feature column, there must be other cases when an average score of infinity or NaN is get. @justinormont can you provide more examples?

[justinormont]

I think, if the average is +/-Infinity, GetIndexClosestToAverage() should return the fold index of a fold with the infinity value. It's a slight better answer for GetIndexClosestToAverage() than returning 0 as that would push down the first fold instead of the closest.

No need to throw an error. Positive/negative infinity are expected values for some metrics. I most often see infinities in log-loss reduction.

[LittleLittleCloud]

@justinormont Thanks for pointing that out. @antoniovs1029 so the only thing that needs to be changed is GetIndexClosestToAverage(). Could you get a quick fix for that? I can also take it if your bandwidth is full

[antoniovs1029]

@LittleLittleCloud If you're able to do this, I'd appreciate it 😄 since I have some other things on my plate right now.