.Net 7 + 8 Garbage Collection Regression Issue

[crollins7]

Description
I had a .Net 6 app that would experience a very slow memory leak. Due to this behavior I set the app pool to automatically recycle every 2 hours. I upgraded the app to .Net 7 with hopes that would resolve the issue and instead the memory leak exploded (14 Gb's/Hour). When I would do a memory dump and analyze I noticed that almost all of the memory is listed as "Free". I recently upgraded the app to .Net 8 and the memory leak persists.

Following @Maoni0's blog post https://maoni0.medium.com/is-the-regression-in-gc-or-something-else-38f10018dd21, I referenced clrgc.dll in the runtime config and the symptoms are significantly relieved which I believe indicates a regression in GC starting in .Net 7.

Configuration

• .Net 8
• Windows Server 2016 14393.6351
• x64
• VM with 8x Core Xeon 4309Y and 32 Gb's of Ram

Regression
As described above there was a slower memory leak originally on .Net 6 and then in .Net 7 and .Net 8 I experienced an insane regression.

Data
Additional Stats gathered when I was on .Net 7 without clrgc.dll

• GC committed bytes increases like crazy throughout the apps runtime.
• GC Fragmentation > 95%
• GC Heap Size seems relatively stable around 1 GB
• The large fragmented memory blocks are all System.Byte[]

The Link Below has Top Level GC Traces and CPU Sample Traces both taken for 1 hour with and without clrgc.dll referenced.
https://drive.google.com/file/d/1nJ8lROr4pYN4IQf4E4KoQDxBN0M5XdfA/view?usp=sharing

This highlights the performance impact experienced with and without clrgc.dll referenced in runtime settings.

Analysis
I have no clue the relevant code that may cause this. I am happy to do any additional testing that would be useful in helping to identify the cause.

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description
I had a .Net 6 app that would experience a very slow memory leak. Due to this behavior I set the app pool to automatically recycle every 2 hours. I upgraded the app to .Net 7 with hopes that would resolve the issue and instead the memory leak exploded (14 Gb's/Minute). When I would do a memory dump and analyze I noticed that almost all of the memory is listed as "Free". I recently upgraded the app to .Net 8 and the memory leak persists.

Following @Maoni0's blog post https://maoni0.medium.com/is-the-regression-in-gc-or-something-else-38f10018dd21, I referenced clrgc.dll in the runtime config and the symptoms are significantly relieved which I believe indicates a regression in GC starting in .Net 7.

Configuration

• .Net 8
• Windows Server 2016 14393.6351
• x64
• VM with 8x Core Xeon 4309Y and 32 Gb's of Ram

Regression
As described above there was a slower memory leak originally on .Net 6 and then in .Net 7 and .Net 8 I experienced an insane regression.

Data
Additional Stats gathered when I was on .Net 7 without clrgc.dll

• GC committed bytes increases like crazy throughout the apps runtime.
• GC Fragmentation > 95%
• GC Heap Size seems relatively stable around 1 GB
• The large fragmented memory blocks are all System.Byte[]

The Link Below has Top Level GC Traces and CPU Sample Traces both taken for 1 hour with and without clrgc.dll referenced.
https://drive.google.com/file/d/1nJ8lROr4pYN4IQf4E4KoQDxBN0M5XdfA/view?usp=sharing

This highlights the performance impact experienced with and without clrgc.dll referenced in runtime settings.

Analysis
I have no clue the relevant code that may cause this. I am happy to do any additional testing that would be useful in helping to identify the cause.

Author:	crollins7
Assignees:	-
Labels:	tenet-performance, area-GC-coreclr
Milestone:	-

[En3Tho]

Just curious: what was the heap fragmentation with .Net6 / clrgc?

Also, does memory leak result in oom? Or it's just "GC takes more memory because it can"? If last then have you tried explicit memory limits?

[crollins7]

Unfortunately when I was on .Net 6 I did not collect those stats before going to .Net 7. That was also 6 months ago so I can't easily recreate it. I can definitively say the memory leak exploded when I went to .Net 7.

The memory leak will result in OOM which is why I have IIS recycle the app pool on a 2 hour schedule. Below is the same screenshot of above with an additional 10 minutes worth of time showing that when I do an app pool recycle I free up the ballooned ram.

I have not tried an explicit memory limit. clrgc.dll being referenced in runtime settings has been my workaround for the past 2 months as it provides significant relief.

[En3Tho]

I wonder what your app does. It's quite curious that your gc handle count grows indefinitely.
Net8 gc is doing gen1 collections more aggressively but it's quite obvous that gen2 is the problem here as it grows really fast and it's survival rate is almost 100%. Do you have caches that hold those objects forever?

[crollins7]

The application is an MVC web app utilizing EF Core 8. There is some usage of minimal API and there are also hangfire jobs (both recurring and background enqueued) that run from the same web app. There is light usage of Memory Cache within the app.

The top level GC traces and CPU sample traces in the opening post may be helpful to someone who has a deeper understanding of their interpretation then I do.

[cshung]

Which process is the process of interest in the trace?

[crollins7]

Which process is the process of interest in the trace?

For the traces with clrgc.dll enabled the PID is 6108. For the traces without PID enabled the PID is 1700.

[markoweb2]

I believe I just wasted quite a few hours with the exact same issue. After upgrading to .NET 8, I noticed my app did not release memory after a large data export task like it used to with .NET 6. Spent a lot of hours going over my code, maybe I made a change that caused a leak or something....
Finally I tried a simple proof of concept console application and confirmed that it is indeed a GC regression that was introduced with .NET 7.

Here is a super simple console application:

public static void Main(string[] args)
{
    var list = new List<string>();

    for (int i = 0; i < 20000000; i++)
    {
        list.Add(i.ToString());
    }

    Console.WriteLine("done");
    list.Clear();
    list = null;
    GC.Collect();

    Console.ReadLine();
}

Here is what the memory usage is supposed to look like with .NET 6:

And here is the memory usage with .NET 7/8:

(this is all in release mode, you can simply change the Target Framework, recompile and see immediate results)
The memory does not get released no matter how long you wait or how many GC.Collect() calls you do.

If I output GC.GetTotalMemory(false) before and after the GC.Collect(), then it is clear that the GC does free up the memory, but it would appear that it doesn't allow the OS to actually reclaim the memory from the process.

Please fix this regression!

[crollins7]

Great find @markoweb2! I just tested your sample code with .net 6, .net 7, and .net 8 and the results are that > .Net 6 does not fully release the memory. I suspect the root cause affects your sample as well as my production code.

I am very interested to test when there is a fix.

[Maoni0]

apologies for the delay. and really sorry this is causing problems😥 I just got back from vacation and took a look at the simple repro that @markoweb2 provided. I haven't looked at the traces @crollins7 provided - I'll check with @cshung if he's looked at them tomorrow. I'll give an update on that.

with @markoweb2's repro I'm seeing with segments (what GC uses in 6.0) after the induced GC the committed size for the process is ~40mb whereas with regions (what GC uses in 7.0+) it's ~800mb. it's worth pointing out that if you don't do this induced GC, the committed sizes are very similar. so I should ask, are you inducing GCs in your production scenario? if so this would explain the difference otherwise it may not.

what causes this big difference after the induced GC has to do with the fact regions does more management of the memory it's already got than segments. with segments, especialy with segments in LOH, if they are empty, we just decommit the whole thing whereas regions would hold onto them and decide if we should decommit some, all or none. and the way it decides this is by looking at various factors, most predominantly the allocation budget which generally indicates how much you'd be allocating before a GC is done to collect it. it makes sense to keep so we don't need to decommit it and have to re-commit it.

the more equivalent thing to do in 8.0 is to induce a GC with the GCCollectionMode.Aggressive mode. however I just discovered a bug that causes part of what GC retains to not be returned which I'm fixing via this PR. without the fix I'm seeing ~100mb, with the fix it's ~10mb. less than with segments because this is a mode that you use to tell the GC you will not be making allocations for a while so GC is letting go everything it can let go (barring the bug). so
I'm not suggesting that you replace everywhere you call GC.Collect() with GC.Collect (2, GCCollectionMode.Aggressive). however, it does make me think we should at least make GC.Collect() behavior similar for regions as we did with segments.

[crollins7]

Thank you for providing insight into what is happening under the hood as well as the PR you submitted.

I did review my code, and I can confirm that I do not call GC.Collect anywhere.

[Maoni0]

I just took a brief look at the traces you provided (thanks so much for collecting these!) and I have a guess what's causing what you saw. most likely it has to do with pinning. there's a lot of pinning going on and there's a known problem with regions for this case that we knew from our internal testing but I just haven't had time to fix. I'll look more in detail to confirm if that's the case (and obviously provide a fix if that is indeed the case). meanwhile, if you wouldn't mind collecting a trace that shows where the pinning comes from, that'd be great! you could use this cmdline -

perfview /nogui /KernelEvents=Process+Thread+ImageLoad /ClrEvents:GC+Stack+GCHandle /clrEventLevel=Informational /MaxCollectSec:1800 /BufferSize:3000 /CircularMB:3000 collect