CrossValidation RegressionMetrics has two more entries than number of folds

[jwood803]

I might be incorrect on this, but wouldn't the number of RegressionMetrics be the same as the number of folds specified in the CrossValidator?

System information

• Windows 10
• .NET Version (eg., dotnet --info):

.NET Command Line Tools (2.1.202)

Product Information:
 Version:            2.1.202
 Commit SHA-1 hash:  281caedada

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17134
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.1.202\

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.9
  Build    : 1632fa1589b0eee3277a8841ce1770e554ece037

Issue

• What did you do?
  Ran cross validation against a pipeline.
• What happened?
  The RegressionMetrics count is two more than the number of folds specified.
• What did you expect?
  The RegressionMetrics count to be the same as the number of folds.

Source code / logs

var pipeline = new LearningPipeline
{
    new TextLoader(dataset).CreateFrom<SalaryData>(useHeader: true, separator: ','),
    new ColumnConcatenator("Features", "YearsExperience"),
    new GeneralizedAdditiveModelRegressor()
};

var crossValidator = new CrossValidator()
{
    Kind = MacroUtilsTrainerKinds.SignatureRegressorTrainer,
    NumFolds = 2
};
var crossValidatorOutput = crossValidator.CrossValidate<SalaryData, SalaryPrediction>(pipeline);

crossValidatorOutput.RegressionMetrics.ForEach(m => Console.WriteLine(m.Rms));

Looking at sklearn, it seems to have the same number of results as the number of folds:

from sklearn.model_selection import cross_val_score

cross_val_score(lin_reg, train_set, train_labels, cv=3)

array([0.96044449, 0.97351702, 0.92777218])

[Zruty0]

I think this is by design. I might be wrong, but I suspect that the extra two metric rows could be the 'overall' metrics and their standard deviation?

Maybe @yaeldekel can confirm or deny.

[codemzs]

@jwood803 @Zruty0 This is by design. The first two rows contain average and std. deviations of the metrics from all the folds.

[Ivanidzo4ka]

Not @yaeldekel , but can confirm.
I think one good solution is to expose "Fold Index" field which getting lost, or maybe just remove this rows from returning results.

[Ivanidzo4ka]

@jwood803 Would you like to have these average and std deviations of metrics in returning result with a way to distinguish them from fold results, or you fine with just fold results?

[jwood803]

@Ivanidzo4ka Maybe it would be best to have them distinguishable from the fold results? My thinking is that that would be easier for discoverability and readability.