Closed
Description
Hey folks
I just try the Recommendation experiment API in the master branch. After I training for a period of time (like 300 sec), I got this error.
Unhandled Exception: System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: SMAC sweeper localSearch threw exception ---> System.MethodAccessException: Attempt by method 'Microsoft.ML.AutoML.SmacSweeper.ComputeForestStats(Double[][])' to access method 'Microsoft.ML.Trainers.FastTree.VectorUtils.GetMean(Double[])' failed.
at Microsoft.ML.AutoML.SmacSweeper.ComputeForestStats(Double[][] leafValues)
at Microsoft.ML.AutoML.SmacSweeper.LocalSearch(ParameterSet parent, FastForestRegressionModelParameters forest, Double bestVal, Double epsilon, Boolean isMetricMaximizing)
--- End of inner exception stack trace ---
at Microsoft.ML.AutoML.SmacSweeper.LocalSearch(ParameterSet parent, FastForestRegressionModelParameters forest, Double bestVal, Double epsilon, Boolean isMetricMaximizing)
at Microsoft.ML.AutoML.SmacSweeper.GreedyPlusRandomSearch(ParameterSet[] parents, FastForestRegressionModelParameters forest, Int32 numOfCandidates, IEnumerable`1 previousRuns)
at Microsoft.ML.AutoML.SmacSweeper.GenerateCandidateConfigurations(Int32 numOfCandidates, IEnumerable`1 previousRuns, FastForestRegressionModelParameters forest)
at Microsoft.ML.AutoML.PipelineSuggester.SampleHyperparameters(MLContext context, SuggestedTrainer trainer, IEnumerable`1 history, Boolean isMaximizingMetric)
at Microsoft.ML.AutoML.PipelineSuggester.GetNextInferredPipeline(MLContext context, IEnumerable`1 history, DatasetColumnInfo[] columns, TaskKind task, Boolean isMaximizingMetric, CacheBeforeTrainer cacheBeforeTrainer, IEnumerable`1 trainerWhitelist)
at Microsoft.ML.AutoML.Experiment`2.Execute()
at Microsoft.ML.AutoML.ExperimentBase`2.Execute(ColumnInformation columnInfo, DatasetColumnInfo[] columns, IEstimator`1 preFeaturizer, IProgress`1 progressHandler, IRunner`1 runner)
at Microsoft.ML.AutoML.ExperimentBase`2.Execute(IDataView trainData, ColumnInformation columnInformation, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)
at Microsoft.ML.ModelBuilder.AutoMLEngine.<>c__DisplayClass23_0`4.<InnerTrainModelAsync>b__4() in C:\Users\xiaoyuz\Source\Repos\machinelearning-tools\Microsoft.ML.ModelBuilder.AutoMLService\AutoMLEngineService\AutoMLEngine.cs:line 277
at System.Threading.Tasks.Task`1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Microsoft.ML.ModelBuilder.AutoMLEngine.<InnerTrainModelAsync>d__23`4.MoveNext() in C:\Users\xiaoyuz\Source\Repos\machinelearning-tools\Microsoft.ML.ModelBuilder.AutoMLService\AutoMLEngineService\AutoMLEngine.cs:line 281
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Microsoft.ML.ModelBuilder.AutoMLEngine.<StartTrainingAsync>d__9.MoveNext() in C:\Users\xiaoyuz\Source\Repos\machinelearning-tools\Microsoft.ML.ModelBuilder.AutoMLService\AutoMLEngineService\AutoMLEngine.cs:line 107
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
at ConsoleApp1.Program.Main(String[] args) in C:\Users\xiaoyuz\Source\Repos\machinelearning-tools\Microsoft.ML.ModelBuilder.AutoMLService\Program.cs:line 36
the dataset I use is this one
I guess it's because of value error (like nan). But I'm not too sure about that, Please take a look, any suggestion will be helpful. Thanks
A minimal reproducable example here
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Metadata
Assignees
Labels
No labels