Fix Recursive Parsing for PFI

[ericjohannsen]

System Information (please complete the following information):

• Windows 11
• ML.NET Version 1.7, AutoML 0.19.0
• .NET Version: .NET Core 3.1

Describe the bug
ArgumentNullException while attempting to retrieve PFI
The model provided does not have a compatible predictor (Parameter 'lastTransformer')

Stack trace
at Microsoft.ML.Runtime.Contracts.CheckValue[T](IExceptionContext ctx, T val, String paramName, String msg)
at Microsoft.ML.PermutationFeatureImportanceExtensions.PermutationFeatureImportance[TMetric,TResult](IHostEnvironment env, ITransformer model, IDataView data, Func1 resultInitializer, Func2 evaluationFunc, Func3 deltaFunc, Int32 permutationCount, Boolean useFeatureWeightFilter, Nullable1 numberOfExamplesToUse)
at Microsoft.ML.PermutationFeatureImportanceExtensions.PermutationFeatureImportance(RegressionCatalog catalog, ITransformer model, IDataView data, String labelColumnName, Boolean useFeatureWeightFilter, Nullable`1 numberOfExamplesToUse, Int32 permutationCount)

To Reproduce

Not sure how best to create a minimal reproducible example including data, but here's the core of what I did (based on #5934)

MLContext mlContext = new MLContext();

ColumnInferenceResults columnInference = mlContext.Auto().InferColumns(trainDataPath, "Label", groupColumns: false);

TextLoader textLoader = mlContext.Data.CreateTextLoader(columnInference.TextLoaderOptions);
IDataView trainDataView = textLoader.Load(trainDataPath);
IDataView testDataView = textLoader.Load(testDataPath);

IEstimator<ITransformer> preFeaturizer = 
	mlContext.Transforms.Categorical.OneHotEncoding(
		new T().GetOneHotInputColumnNames().Select(_ => new InputOutputColumnPair(_)).ToArray()
		);

ColumnInformation columnInformation = columnInference.ColumnInformation;
columnInformation.IgnoredColumnNames.AddIfMissing("Foo");
columnInformation.CategoricalColumnNames.Remove("Foo");
columnInformation.NumericColumnNames.Remove("Foo");

BinaryExperimentSettings experimentSettings = new BinaryExperimentSettings()
{
	MaxExperimentTimeInSeconds = experimentTime,
	OptimizingMetric = BinaryClassificationMetric.F1Score
};

var experiment = mlContext.Auto().CreateBinaryClassificationExperiment(experimentSettings);

ExperimentResult<BinaryClassificationMetrics> experimentResult = experiment.Execute(trainDataView, columnInformation, preFeaturizer, progress);

RunDetail<BinaryClassificationMetrics> bestRun = experimentResult.BestRun;

// Exception thrown here
var permutationFeatureImportance = mlContext
	   .Regression
	   .PermutationFeatureImportance(bestRun.Model, testDataViewWithBestScore, permutationCount: 3);

The LastTransformer of bestRun.Model is BinaryPredictionTransformer<Microsoft.ML.Calibrators.CalibratedModelParametersBase<Microsoft.ML.Trainers.lightGbm.LightGbmBinaryModelParameters, ...>>

Expected behavior
Retrieve the PFI information.

[michaelgsharp]

Hey @ericjohannsen,

This issue was actually fixed just yesterday by pr #6085. You can try building the latest from main and see if that fixes your issue, or you can the latest preview package here, https://pkgs.dev.azure.com/dnceng/public/_packaging/MachineLearning/nuget/v3/index.json.

Closing this issue for now as it should be resolved, but please re-open the issue if it isn't resolved.

[IntegerMan]

Is there any further details on the preview package route or any estimate when a released package might be available? I added that JSON as a NuGet package source, but the latest available version of AutoML and Microsoft.ML from that source match the latest on NuGet and both have this issue as well.

I had hoped to include explainability in a user group talk / workshop I'm giving Thursday on AutoML, but it looks like that might be too optimistic if I wanted folks to be able to follow along.

[ericjohannsen]

@michaelgsharp @IntegerMan
I was able to reference Microsoft.ML 2.0.0-preview.22115.2 and Microsoft.ML.AutoML 0.20.0-preview.22115.2 using the source on Azure, but still encounter the same problem.

<PackageReference Include="Microsoft.ML" Version="2.0.0-preview.22115.2" />
<PackageReference Include="Microsoft.ML.AutoML" Version="0.20.0-preview.22115.2" />

[ericjohannsen]

@michaelgsharp Any word on this?

[michaelgsharp]

In the preview version instead of passing in bestRun.Model.LastTransformer, you should just pass in bestRun.Model. The PFI code should automatically pull out the transformer that it needs. Can you try doing that and let me know what happens? I'll repoen this issue for now.

We are planning on doing a servicing release which will include this bug fix next week as well.

[ericjohannsen]

@michaelgsharp I was already passing in bestRun.Model

var bestModel = bestRun.Model;

var permutationFeatureImportance = mlContext
	   .Regression
	   .PermutationFeatureImportance(bestRun.Model, testDataViewWithBestScore, permutationCount: 3);

This issue has been automatically marked no-recent-activity because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will remove no-recent-activity.

[michaelgsharp]

@ericjohannsen the servicing release is out with the fix for this. Can you try with the latest 1.7.1 and let me know if it solves your issue?

[ericjohannsen]

@michaelgsharp No joy with 1.7.1. If it helps:

Let me know what other information might be helpful diagnosing this.