Added support for RankingMetrics with CrossValSummaryRunner

[mstfbl]

Fix #5381

Previously, AutoFitRankingTest was never testing RankingMetrics with CrossValSummaryRunner due to the row limit for cross validation testing, and as the MLSR dataset that trainDataView is pulling from is large:

machinelearning/test/Microsoft.ML.AutoML.Tests/AutoFitTests.cs

Lines 123 to 141 in f87a3bb

	[LightGBMFact]
	public void AutoFitRankingTest()
	{
	string labelColumnName = "Label";
	string scoreColumnName = "Score";
	string groupIdColumnName = "GroupId";
	string featuresColumnVectorNameA = "FeatureVectorA";
	string featuresColumnVectorNameB = "FeatureVectorB";
	var mlContext = new MLContext(1);
	// STEP 1: Load data
	var reader = new TextLoader(mlContext, GetLoaderArgsRank(labelColumnName, groupIdColumnName, featuresColumnVectorNameA, featuresColumnVectorNameB));
	var trainDataView = reader.Load(new MultiFileSource(DatasetUtil.GetMLSRDataset()));
	var testDataView = mlContext.Data.TakeRows(trainDataView, 500);
	trainDataView = mlContext.Data.SkipRows(trainDataView, 500);
	// STEP 2: Run AutoML experiment
	var experiment = mlContext.Auto()
	.CreateRankingExperiment(5);

machinelearning/src/Microsoft.ML.AutoML/API/ExperimentBase.cs

Lines 109 to 130 in 6bae29f

	public ExperimentResult<TMetrics> Execute(IDataView trainData, ColumnInformation columnInformation,
	IEstimator<ITransformer> preFeaturizer = null, IProgress<RunDetail<TMetrics>> progressHandler = null)
	{
	// Cross val threshold for # of dataset rows --
	// If dataset has < threshold # of rows, use cross val.
	// Else, run experiment using train-validate split.
	const int crossValRowCountThreshold = 15000;
	var rowCount = DatasetDimensionsUtil.CountRows(trainData, crossValRowCountThreshold);
	var samplingKeyColumnName = GetSamplingKey(columnInformation?.GroupIdColumnName, columnInformation?.SamplingKeyColumnName);
	if (rowCount < crossValRowCountThreshold)
	{
	const int numCrossValFolds = 10;
	var splitResult = SplitUtil.CrossValSplit(Context, trainData, numCrossValFolds, samplingKeyColumnName);
	return ExecuteCrossValSummary(splitResult.trainDatasets, columnInformation, splitResult.validationDatasets, preFeaturizer, progressHandler);
	}
	else
	{
	var splitResult = SplitUtil.TrainValidateSplit(Context, trainData, samplingKeyColumnName);
	return ExecuteTrainValidate(splitResult.trainData, columnInformation, splitResult.validationData, preFeaturizer, progressHandler);
	}
	}

As a result, this lacking of support for RankingMetrics with CrossValSummaryRunner was not noticed until Issue #5381. This PR adds support for this case, and edits the RankingMetrics AutoML unit tests to include testing of RankingMetrics with CrossValSummaryRunner as well.

[codecov]

Codecov Report

Merging #5386 into master will increase coverage by 0.06%.
The diff coverage is 61.66%.

@@            Coverage Diff             @@
##           master    #5386      +/-   ##
==========================================
+ Coverage   74.02%   74.09%   +0.06%     
==========================================
  Files        1019     1019              
  Lines      190308   190355      +47     
  Branches    20462    20469       +7     
==========================================
+ Hits       140877   141039     +162     
+ Misses      43888    43789      -99     
+ Partials     5543     5527      -16     

Flag	Coverage Δ
#Debug	74.09% <61.66%> (+0.06%)	⬆️
#production	69.86% <48.78%> (+0.08%)	⬆️
#test	87.71% <89.47%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...AutoML/Experiment/Runners/CrossValSummaryRunner.cs	56.73% <0.00%> (-7.78%)	⬇️
...soft.ML.Data/DataLoadSave/DataOperationsCatalog.cs	85.50% <ø> (ø)
src/Microsoft.ML.TimeSeries/STL/InnerStl.cs	90.50% <0.00%> (ø)
src/Microsoft.ML.TimeSeries/STL/Loess.cs	84.61% <0.00%> (ø)
src/Microsoft.ML.TimeSeries/Deseasonality.cs	92.42% <50.00%> (+4.18%)	⬆️
...crosoft.ML.TimeSeries.Tests/TimeSeriesDirectApi.cs	99.41% <85.71%> (-0.29%)	⬇️
...rosoft.ML.TimeSeries/SrCnnEntireAnomalyDetector.cs	91.85% <100.00%> (+0.22%)	⬆️
test/Microsoft.ML.AutoML.Tests/AutoFitTests.cs	93.79% <100.00%> (+0.02%)	⬆️
...c/Microsoft.ML.FastTree/Utils/ThreadTaskManager.cs	79.48% <0.00%> (-20.52%)	⬇️
...icrosoft.ML.AutoML/Experiment/SuggestedPipeline.cs	88.65% <0.00%> (-4.13%)	⬇️
... and 13 more

[LittleLittleCloud]

LGTM