Let ML.NET cancel unmanaged code.

[andrasfuchs]

System Information:

• OS & Version: Windows 11 [Version 10.0.22621.819]
• ML.NET Version: ML.NET v2.0.0
• .NET Version: .NET 7.0.100

Describe the bug
I run an ML.NET experiment for 10 hours. I limited the training for FastTree and FastForest algorithms and I made sure that every new best model is saved even if the whole run fails. I locked Windows and let it run during the night.
When I checked it in the morning (12 hours after the experiment started) I couldn't even log in, I had to do a power off-power on to get access. When I checked the logs it looked like the last experiment was saved after ~5 hours of running.
The dataset wasn't particularly big this time, it was only a 205 MB CSV file. My system has 32 GB RAM, and I set the maximum memory usage for the experiment to 20 GB.

To Reproduce
Steps to reproduce the behavior:

1. Create a pipeline that includes only FastTree and FastForest, for example:

var pipeline =
  mlContext.Auto().Featurizer(trainTestData.TrainSet, numericColumns: new[] { "Features" })
      .Append(mlContext.Auto().Regression(useFastTree: true, useLbfgs: false, useSdca: false, useFastForest: true, useLgbm: false));

2. Create an ML.NET experiment and set its memory limit. I set it to 20 GB, because I usually have ~24 GB available when I start the training.

var experiment = mlContext.Auto().CreateExperiment();

experiment
    .SetPipeline(pipeline)
    .SetTrainingTimeInSeconds(trainingTimeInSeconds)
    .SetRegressionMetric(RegressionMetric.RSquared, labelColumn: "Label")
    .SetDataset(trainTestData.TrainSet, trainTestData.TestSet)
    .SetMonitor(monitor)
    .SetMaximumMemoryUsageInMegaByte(20 * 1024);

4. Run the experiment for at least 10 hours
5. Check if you can still use the system after 5+ hours of running

Expected behavior
I would expect the experiment to run without a system-freezing halt.

Screenshots, Code, Sample Projects
The 205 MB CSV dataset can be downloaded from here:
BBD_20221122__TrainingData.Sleep.MLP12__MLP12_0p25Hz-250Hz__Session_SegmentedData_Sleep_Level__15782rows.zip

Additional context
My project is open-source so both the source and the data are available. Let me know if you need them for testing.

[andrasfuchs]

I run further tests, and I think I found the root cause of this problem: the LightGbmRegression estimator can't be cancelled during it's startup and it crashes if the systems runs out of virtual memory.

Here are my logs:

11:32:57.1 info:     Running Trial #  16
11:33:03.1 info:   Resources Trial #  16 - Estimator: LightGbmRegression                  -   Memory:        2,2 GB -      CPU:         41,8% - Duration:   0,1 minutes
11:33:06.0 info:   Completed Trial #  16 - Estimator: LightGbmRegression                  -   Metric:        0,6090 - Duration:   0,1 minutes
11:33:06.0 info:  ------------------------------------------------------------------------------------------------------------------------------------------------------
11:33:06.0 info:    New Best Trial #  16 - Estimator: LightGbmRegression                  -   Metric:        0,6090
11:33:06.0 info:  ------------------------------------------------------------------------------------------------------------------------------------------------------
11:33:06.0 info:     Running Trial #  17
11:33:12.0 info:   Resources Trial #  17 - Estimator: LightGbmRegression                  -   Memory:        2,7 GB -      CPU:         43,8% - Duration:   0,1 minutes
11:33:18.1 info:   Resources Trial #  17 - Estimator: LightGbmRegression                  -   Memory:        6,8 GB -      CPU:         85,5% - Duration:   0,2 minutes
11:33:24.1 info:   Resources Trial #  17 - Estimator: LightGbmRegression                  -   Memory:        6,8 GB -      CPU:         83,2% - Duration:   0,3 minutes
11:33:24.1 warn:   Resources Trial #  17 - Cancellinng trial due to timeout
11:33:24.1 info:      Failed Trial #  17 - Estimator: LightGbmRegression                  - Error: Operation was canceled.
11:33:24.1 info:  ------------------------------------------------------------------------------------------------------------------------------------------------------
11:33:24.1 info:     Running Trial #  18
11:33:27.1 info:   Completed Trial #  18 - Estimator: LbfgsPoissonRegressionRegression    -   Metric:        0,0656 - Duration:   0,1 minutes
11:33:27.1 info:  ------------------------------------------------------------------------------------------------------------------------------------------------------
11:33:27.1 info:     Running Trial #  19
11:33:33.2 info:   Resources Trial #  19 - Estimator: LightGbmRegression                  -   Memory:        2,8 GB -      CPU:         41,5% - Duration:   0,1 minutes
11:33:35.7 info:   Completed Trial #  19 - Estimator: LightGbmRegression                  -   Metric:        0,5047 - Duration:   0,1 minutes
11:33:35.8 info:  ------------------------------------------------------------------------------------------------------------------------------------------------------
11:33:35.8 info:     Running Trial #  20
11:33:41.8 info:   Resources Trial #  20 - Estimator: LightGbmRegression                  -   Memory:        2,9 GB -      CPU:         45,4% - Duration:   0,1 minutes
11:33:47.8 info:   Resources Trial #  20 - Estimator: LightGbmRegression                  -   Memory:       27,5 GB -      CPU:         84,2% - Duration:   0,2 minutes
11:33:53.8 info:   Resources Trial #  20 - Estimator: LightGbmRegression                  -   Memory:       40,1 GB -      CPU:         91,2% - Duration:   0,3 minutes
11:33:53.8 warn:   Resources Trial #  20 - Cancellinng trial due to timeout
11:33:53.9 warn:   Resources Trial #  20 - Cancelling trial due to memory usage
11:33:54.1 warn:   Resources Trial #  20 - Cancelling trial due to low virtual memory
11:33:59.8 info:   Resources Trial #  20 - Estimator: LightGbmRegression                  -   Memory:       46,5 GB -      CPU:          0,0% - Duration:   0,4 minutes
11:33:59.9 warn:   Resources Trial #  20 - Cancellinng trial due to timeout
11:33:59.9 warn:   Resources Trial #  20 - Cancelling trial due to memory usage
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation

I set the cancellation thresholds to

• 15 seconds timeout
• 36+ GB of memory usage
• <2 GB free virtual memory

Please notice that the estimator was cancelled successfully in trials 4, 6, 9, 12,13 and 17. If you look at the last trial (20) you can see that my monitor tries to cancel the estimator again and again without success until the process runs out of virtual memory and it gets the "bad allocation" exception repeatedly and finally crashes the whole process.

[michaelgsharp]

@LittleLittleCloud any thoughts on this? looks like a cancellation error. Unless its a LightGBM issue itself.

[LittleLittleCloud]

LightGBM can't be canceled when it's calling into unmanaged dll because cancellation pattern of ml.net is not implemented there. @michaelgsharp is there a way to cancel a running estimator when it's running into unmanaged code by calling mlcontext.CancelExection()?

[andrasfuchs]

It might a little drastic, but I will try to do some experiments when I forcefully terminate the thread that runs the unmanaged code in my IMonitor event handler. For this I will need to modify the ML.NET part to pass the unmanaged ThreadId in the TrialSettings class, but if it works it would be still better than crashing the whole process because I run out of free virtual memory.

[LittleLittleCloud]

@andrasfuchs Another solution that might be an easier path for you is to use a smaller lightGBM search space. The current default lightGBM search space might be too large to an extend that doesn't actually benefit the final training result.

Here's how to pass a smaller lightGBM search space, which we also apply when running azure benchmark and proves to have little impact to final result

var lgbmDefaultOption = new LgbmOption
{
       LabelColumnName = label,
       FeatureColumnName = featureName,
};
var lgbmSearchSpace = new SearchSpace<LgbmOption>(lgbmDefaultOption);

// use a smaller NumberOfLeaves and NumberOfTrees to avoid OOM
lgbmSearchSpace["NumberOfLeaves"] = new UniformIntOption(4, 1024 * 4, true, 4);
lgbmSearchSpace["NumberOfTrees"] = new UniformIntOption(4, 1024 * 4, true, 4);

// apply new lgbmSearchSpace
var pipeline = context.Transforms.Conversion.MapValueToKey(label, label)
                                        .Append(context.Auto().Featurizer(trainData, columnInformation, featureName))
                                        .Append(context.Auto().MultiClassification(labelColumnName: label, featureColumnName: featureName, lgbmSearchSpace: lgbmSearchSpace))

[michaelgsharp]

@LittleLittleCloud looks like the work item for this should be to add that capability when we are calling into unmanaged code.