Memory leak

[ADD-Eugenio-Lopez]

We want to send files using a web application on asp.net core with a kestrel. We have programmed a controller that responds to httpget and returns files that we keep in a MemoryCache.
When the MemoryCache was loaded, we made several requests for the 162 files it contains and observed that the memory of the process does not stop growing at each request.
We have implemented the Response.OnComplete to Dispose the MemoryStream used and make a GarbageCollector, with this we have improved the use of memory and now the increase in the use of memory is about 300 KB per request. This is a problem in case of thousands of requests.
We have executed the performance of VisualStudio and making comparisons between each call, it does not show differences in the use of memory but in the TaskManager the process does not stop growing.
Any idea what the problem might be?

Here is our file to send files:

    /// <summary> Gets a file from cache. </summary>
    /// <param name="fileName"> Filename of the file. </param>
    /// <returns> The file to return. </returns>
    private ActionResult GetFile(string fileName)
    {
        ICacheEntry memoryEntry;

        if (this.memoryCache.TryGetValue(fileName, out memoryEntry))
        {
            // Gets original stream from the cache.
            Stream originalStream = (Stream)memoryEntry.Value;

            Stream stream = new MemoryStream();

            // Copy stream to new one because after send to client it is closed.
            originalStream.CopyTo(stream);

            // Restart positions of streams.
            originalStream.Seek(0, SeekOrigin.Begin);
            stream.Seek(0, SeekOrigin.Begin);

            // Configures the event OnCompleted request to dispose stream object.
            Response.Headers["Content-Disposing"] = "attachment";

            Response.OnCompleted(() =>
            {
                stream.Dispose();
                stream = null;

                // Calls Garbage Collector to free memory, in other case it is not freed.
                System.GC.Collect();

                return Task.CompletedTask;
            });

            // The content type parameter depends on file type.
            if (fileName.EndsWith(".application"))
            {
                return new FileStreamResult(stream, "application/x-ms-application");
            }

            if (fileName.EndsWith(".manifest"))
            {
                return new FileStreamResult(stream, "application/x-ms-manifest");
            }

            return new FileStreamResult(stream, "application/octet-stream");
        }

        return null;
    }

[davidfowl]

I have lots of questions/comments:

• Who populates this cache?

• Did you try using a memory profiler (like the one in visual studio or dotMemory)?

• Why are you caching Streams instead of bytes? Seems like you're copying into a new MemoryStream for every time.

• How big are the files?

• Disposing a memory stream doesn't really do anything.

• You shouldn't be calling System.GC.Collect() anywhere.

• You can set a size limit on the memory cache if you're storing bytes.

[ADD-Eugenio-Lopez]

We will try to answer each question in order:

• We populate this cache at the beginning and we do not want its content to change with the multiple requests.
• We use the Visual studio memory profile and we have not seen significant differences that indicate where the memory leaks are.
• We are copying the MemoryStream each time because at the end of each request close send stream and the next time we cannot acces to the content of the stream. We will try to use bytes instead of streams to see the result.
• Until 660 KB.
• We know that it does not do anything to dispose of the stream, we were doing memory profile of the process and we tried to see if it influences the process.
• We know that we shouldn't be calling System.GC.Collect() anywhere, but we were testing the use of memory that the process really used. One question: When is System.GC.Collect() thrown? because we are making multiple requests and the memory always grows and we do not see when the memory is released.
• We will try to set limit on memory cache.

[davidfowl]

We are copying the MemoryStream each time because at the end of each request close send stream and the next time we cannot acces to the content of the stream. We will try to use bytes instead of streams to see the result.

Don't, just cache the byte[] and avoid the copy the convoluted code to copy and dispose streams.

[ADD-Eugenio-Lopez]

We change all streams by byte[] and the memory usage has stabilized.
Thanks for your answers.

[ADD-Eugenio-Lopez]

In our last test we have eliminated the call to System.GC.Collect() and we have launched several requests until the server process reaches 100MB, we are monitoring it but it never decreases in size, when we had the System.GC.Collect() put in code it remained at a size around 48MB.
When is the System.GC.Collect() applied on its own?
Do you have to configure something to do it?

[dcarr42]

You can turn off ServerGC if you want GC to happen more frequently. But I'm not sure I would recommend it in your scenario.

[ADD-Eugenio-Lopez]

We want limit the size of ".Net Core Host" process.

[dcarr42]

So maybe your use case is genuine. Triggering a collection is not a complete disaster. Also applying a size limit as @davidfowl mentioned.

[ADD-Eugenio-Lopez]

The size of the memory cache does not grow because we have it controlled, what grows is the size of the process observed in the TaskManager and it seems that it never executes the GC. We want to control the size of the process.

[dcarr42]

Are you getting the same memory issues when running locally inside VS? Some profiling will assess whether your memory usage is steady under nominal conditions. But to reiterate GC is not predictable and can not be relied on.

[ADD-Eugenio-Lopez]

We test our server with VS and we launch the memory profile, after each request the quantity of objects and use of memory are the same. If we call GC in our code, in the Task Manager the process every time uses the same memory, but if we delete de call to GC the memory rises in each request.

[davidfowl]

cc @Maoni0

[Maoni0]

In our last test we have eliminated the call to System.GC.Collect() and we have launched several requests until the server process reaches 100MB, we are monitoring it but it never decreases in size, when we had the System.GC.Collect() put in code it remained at a size around 48MB.

if you could help me understand this in a bit more detail that'd be great.

• if you don't have GC.Collect() at all, the memory usage reaches 100mb and then stays there?
• do you have any sort of memory limitation set on the process?
• what's the total amount of the memory on the machine if there's no mem limit?
• since you are using Server GC, how many cores is the process using?
• what version of coreclr are you using? if you are using 2.2 would it be possible to try out a 3.0 build with a hard limit config set?
• finally if we need to debug this, would it be possible to get a perf trace (you can use eventpipe in 3.0 on both Windows and Linux; and if you are on Windows & older version of coreclr you can use ETW)

[ADD-Eugenio-Lopez]

Thanks for your answers and your time.

• The server where we want to deploy has 20 Cores and 40GB of memory at our disposal so they are not limiting.
• We do not have limitation of memory process, but we have a warning from IT that if a process consumes a lot of resources it is killed by the system.
• We are also concerned that the process grows disproportionately and consumes all the resources of the server where it is deployed.
• We are using netcoreapp2.2.
• We have performed a functional test using the GC and without using it and the debugging result is the one that can be seen in the attached images.
  
  
  In the first capture the process consumes memory in each request and never freed, in the second capture the process call GC after each request and the process memory is ever the same.

[Maoni0]

thanks for the info @urodolf. what I'm seeing from the 2 charts you have is there's not a whole lot of difference between the LDS (live data size) (which is what you have in the 2nd chart) of ~300mb and the max mem in the 1st chart which is ~470mb. that's about 8.5mb of allocation per heap. I'd imagine each heap allows you allocate more than that before it triggers a GC. so if say your gen0 alloc budget is 15mb, it means you will see approx. 600mb before a GC happens. so if you monitor it long enough at some point a GC will happen after enough allocations. if you are on windows you can collect ETW events.

now, for restricting the memory usage, first of all, having only (470-300=170mb)/300 =56% is not a lot of headroom for GCs to run. either you need to accommodate allocations beyond your LDS or you do not allocate during your requests and the latter is very unlikely. secondly, you can decide how much GC cost you are willing to pay, I would guess triggering a GC for every request is very costly that's not something you'd want in your production. and as I mentioned you could also try out new configs in 3.0; or use the GCHeapCount to limit the # of heaps which means you'll be allowed less allocations before the next GC happens.