Skip to content

[AutoML] Recommendation experiment got SMAC local search exception during training #4354

Closed

Description

Hey folks

I just try the Recommendation experiment API in the master branch. After I training for a period of time (like 300 sec), I got this error.

Unhandled Exception: System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: SMAC sweeper localSearch threw exception ---> System.MethodAccessException: Attempt by method 'Microsoft.ML.AutoML.SmacSweeper.ComputeForestStats(Double[][])' to access method 'Microsoft.ML.Trainers.FastTree.VectorUtils.GetMean(Double[])' failed.
   at Microsoft.ML.AutoML.SmacSweeper.ComputeForestStats(Double[][] leafValues)
   at Microsoft.ML.AutoML.SmacSweeper.LocalSearch(ParameterSet parent, FastForestRegressionModelParameters forest, Double bestVal, Double epsilon, Boolean isMetricMaximizing)
   --- End of inner exception stack trace ---
   at Microsoft.ML.AutoML.SmacSweeper.LocalSearch(ParameterSet parent, FastForestRegressionModelParameters forest, Double bestVal, Double epsilon, Boolean isMetricMaximizing)
   at Microsoft.ML.AutoML.SmacSweeper.GreedyPlusRandomSearch(ParameterSet[] parents, FastForestRegressionModelParameters forest, Int32 numOfCandidates, IEnumerable`1 previousRuns)
   at Microsoft.ML.AutoML.SmacSweeper.GenerateCandidateConfigurations(Int32 numOfCandidates, IEnumerable`1 previousRuns, FastForestRegressionModelParameters forest)
   at Microsoft.ML.AutoML.PipelineSuggester.SampleHyperparameters(MLContext context, SuggestedTrainer trainer, IEnumerable`1 history, Boolean isMaximizingMetric)
   at Microsoft.ML.AutoML.PipelineSuggester.GetNextInferredPipeline(MLContext context, IEnumerable`1 history, DatasetColumnInfo[] columns, TaskKind task, Boolean isMaximizingMetric, CacheBeforeTrainer cacheBeforeTrainer, IEnumerable`1 trainerWhitelist)
   at Microsoft.ML.AutoML.Experiment`2.Execute()
   at Microsoft.ML.AutoML.ExperimentBase`2.Execute(ColumnInformation columnInfo, DatasetColumnInfo[] columns, IEstimator`1 preFeaturizer, IProgress`1 progressHandler, IRunner`1 runner)
   at Microsoft.ML.AutoML.ExperimentBase`2.Execute(IDataView trainData, ColumnInformation columnInformation, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)
   at Microsoft.ML.ModelBuilder.AutoMLEngine.<>c__DisplayClass23_0`4.<InnerTrainModelAsync>b__4() in C:\Users\xiaoyuz\Source\Repos\machinelearning-tools\Microsoft.ML.ModelBuilder.AutoMLService\AutoMLEngineService\AutoMLEngine.cs:line 277
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.ML.ModelBuilder.AutoMLEngine.<InnerTrainModelAsync>d__23`4.MoveNext() in C:\Users\xiaoyuz\Source\Repos\machinelearning-tools\Microsoft.ML.ModelBuilder.AutoMLService\AutoMLEngineService\AutoMLEngine.cs:line 281
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.ML.ModelBuilder.AutoMLEngine.<StartTrainingAsync>d__9.MoveNext() in C:\Users\xiaoyuz\Source\Repos\machinelearning-tools\Microsoft.ML.ModelBuilder.AutoMLService\AutoMLEngineService\AutoMLEngine.cs:line 107
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at ConsoleApp1.Program.Main(String[] args) in C:\Users\xiaoyuz\Source\Repos\machinelearning-tools\Microsoft.ML.ModelBuilder.AutoMLService\Program.cs:line 36

the dataset I use is this one

I guess it's because of value error (like nan). But I'm not too sure about that, Please take a look, any suggestion will be helpful. Thanks

A minimal reproducable example here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions