Skip to content

System.OperationCanceledException after AutoML experiment #5620

Open

Description

System information

.NET SDK (reflecting any global.json):
Version: 5.0.103
Commit: 72dec52dbd

Runtime Environment:
OS Name: Windows
OS Version: 10.0.19042
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\5.0.103\

Host (useful for support):
Version: 5.0.3
Commit: c636bbdc8a

.NET SDKs installed:
5.0.103 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
Microsoft.AspNetCore.All 2.1.25 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.25 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.25 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Issue

After an experiment ended (TimeOut elapsed, with token: CancellationToken.None)
Sometimes the access of experimentResult.BestRun.Estimator.Fit(_exDataView) will raise System.OperationCanceledException.

  • What happened?
 System.OperationCanceledException: Operation was canceled.
      at Microsoft.ML.Runtime.Contracts.CheckAlive(IHostEnvironment env)
  at Microsoft.ML.Transforms.ValueToKeyMappingTransformer.Train(IHostEnvironment env, IChannel ch, ColInfo[] infos, IDataView keyData, ColumnOptionsBase[] columns, IDataView trainingData, Boolean autoConvert)
  at Microsoft.ML.Transforms.ValueToKeyMappingTransformer..ctor(IHostEnvironment env, IDataView input, ColumnOptionsBase[] columns, IDataView keyData, Boolean autoConvert)
  at Microsoft.ML.Transforms.ValueToKeyMappingEstimator.Fit(IDataView input)
  at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
  at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
  at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
  at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
  • What did you expect?

This exception does not occur all the time when accessing BestRun prop.

Source code / logs

var experimentSettings = new MulticlassExperimentSettings
{
    MaxExperimentTimeInSeconds = _maxExperimentTimeInSeconds,
    CancellationToken = pManager.Token, // CancellationToken.None
    OptimizingMetric = MulticlassClassificationMetric.MacroAccuracy
}
var experiment= _mlContext.Auto().CreateMulticlassClassificationExperiment(experimentSettings);
var experimentResult= experiment.Execute(_trainDataView, progressHandler: experimentProgress,
                preFeaturizer: _mlContext.Transforms.DropColumns("LabelFeaturized"));
experimentResult.BestRun.Estimator.Fit(_trainDataView);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    AutoML.NETAutomating various steps of the machine learning processneed infoThis issue needs more info before triage

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions