WeakReference<IHost> memory leak?

[venkateshkrc]

System information

• ML.NET Version: 1.4.0

Issue

• What did you do?

1. Created a pipeline of ImageLoadingEstimator + ImageResizingEstimator + ImagePixelExtractingEstimator + OnnxScoringEstimator.
2. Create a ITransformer object by fitting the pipeline.
3. Performed ITransformer.Transform multiple times.

PS: Creating a PredictionEngine and performing PredictionEngine.Predict seems to lock the image as mentioned in issue 4585

• What happened?

WeakReference objects seems to accumulate.

The number of objects does not change even after performing GC.Collect()

Is this possibly a memory leak?
The application is built is Release configuration.

[mstfbl]

Hey @crvkumar , thank you for letting us know about this possible memory leak. Could you please share with us your code with which you observed this behavior? I will run your code and see if I see an accumulation of WeakReference objects as well as its impact on memory.

[venkateshkrc]

Hello @mstfbl. Thanks for the response.
Created a sample code reproducing this behavior in the below repo. Please check.

https://github.com/crvkumar/ML.NETMemoryLeakSample

[mstfbl]

Hi @crvkumar , I ran the sample code you linked in Visual Studio 2019 with OnnxTransformer v1.3.1. I waited at the 2 parts of the code where the first and second snapshots are made. I did not see a memory leak when I ran it in Debug and Release modes, where process memory stayed stable at 61-62 MB. I also did not see any odd increases in the amount of WeakReference objects. Thus I was not able to reproduce your issue.

Perhaps there were other factors influencing the results you got. Please try running it again and let us know if you consistently see a memory leak. Thanks!

[venkateshkrc]

Hi @mstfbl. Thanks for checking the issue.

I am consistently getting the same results. I have attached a video for your reference.
I was using OnnxTransformer 1.4.0 but versioning down to 1.3.1 didn't change the behavior either.

https://github.com/crvkumar/ML.NETMemoryLeakSample/blob/master/MemoryLeakDemo%20-%20Microsoft%20Visual%20Studio%203_10_2020%2010_39_39%20AM.mp4

[mstfbl]

Hi @crvkumar ,
I see, indeed in your case there is an increase of memory within a short time. I have uploaded my own demonstration showing no increase during one snapshot, and a small but expected increase between the first and second snapshots (The point in time when I iterate from the 1st to the 2nd snapshot is during the middle point of the video, unfortunately the terminal is not shown in this video).

https://github.com/mstfbl/machinelearning/blob/memoryleakinquirydemonstration/demonstration/MemoryLeakDemo%20-%20Microsoft%20Visual%20Studio%20Preview%202020-03-09%2018-58-33.mp4

[venkateshkrc]

Hello @mstfbl Thank you. May I know what version of onnxruntime are you using.?
I guess ML.NET only supports 0.5.1.

[mstfbl]

Correct, I'm running OnnxRuntime v0.5.1 and OnnxTransformer v1.3.1.
Edit: We support OnnxRuntime 1.0 in ML.NET 1.5.0-preview.

[venkateshkrc]

May I know how could you run the project in "Any CPU" config? Does the project is set in "x64"?

[mstfbl]

Of course, the "Any CPU" config on my computer is just set to "x64", so it is effectively running in "x64".

Also, I'd like to make an addition to my earlier comment. We currently support OnnxRuntime 1.0 as seen in this PR for ML.NET 1.5.0-preview. (Thanks @antoniovs1029 for the heads up!)

[venkateshkrc]

OK. Thank you.
Do you have any idea why the behavior is different in the two machines? Do you have any pointers where to look?

[strikene]

I have the same problem. I use 1.50preview2 and 1.4
By the way, even without image prediction, a mere multi-classification of text would leak.
The first picture was when I started the application.

When I started 5 text classification predictions, I could no longer capture snapshots, but I could see that memory was still growing after GC ，

public class ModelMInput
{
    [ColumnName("Action"), LoadColumn(0)]
    public string Action { get; set; }


    [ColumnName("DMString"), LoadColumn(1)]
    public string DMString { get; set; }


}
public class PredictedEngineOutput
{

    [ColumnName("PredictedLabel")]
    public String Prediction { get; set; }
    public float[] Score { get; set; }

    [ColumnName("Action")]
    [KeyType(1)]
    public UInt32 InputAction { get; set; }

}




public class DemoModel
{
    private MLContext _MLContext;
    private DataViewSchema _inputschema;
    private ITransformer _Model;
    public string ModelName { get; set; }
    public string[] TaginModel { get; private set; }
    public DemoModel(string Path)
    {
        _MLContext = new MLContext();
        _Model = _MLContext.Model.Load(Path, out _inputschema);
        var OutputSchema = _Model.GetOutputSchema(_inputschema);
        var column = OutputSchema.GetColumnOrNull("Score");
        var slotNames = new VBuffer<ReadOnlyMemory<char>>();
        column.Value.GetSlotNames(ref slotNames);
        var alltag = slotNames.DenseValues().ToArray();
        TaginModel = new string[alltag.Count()];

        for (int i = 0; i < TaginModel.Length; i++)
        {
            TaginModel[i] = alltag[i].ToString();
        }

        ModelName = new FileInfo(Path).Name;

    }
    public PredictedEngineOutput Predict(string dmstring)
    {
        using (var PredictionEngine = _MLContext.Model.CreatePredictionEngine<ModelMInput, PredictedEngineOutput>(_Model))
        {
            var x = PredictionEngine.Predict(new ModelMInput() { DMString = dmstring });

            return x;
        }
    }
}

[mstfbl]

Hi @crvkumar and @strikene ,
Thank you for following up on this. We're still working on reproducing this issue locally, and will get back to you soon.

[mstfbl]

Hi @strikene ,

I'm attempting to run the sample code you've written. Do you have the dataset where Path is pointing?