In .NET 7 large number of concurrent requests will memory leak

[Cricle]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

When TFM=.NET7
wrk -t8 -c1000 -d30s --latency http://192.168.2.86:8889 many times, memory will grow to 1G.

When TFM=.NET6
wrk -t8 -c1000 -d30s --latency http://192.168.2.86:8889 many times, memory only 145M.

The Code in program

            Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
            var builder = WebApplication.CreateBuilder(args);
            
            builder.Services.AddSingleton(new HitokotoManager(Init()));
            builder.Services.AddResponseCompression();

            var app = builder.Build();

            app.UseResponseCompression();
            app.Use(async (HttpContext builder, RequestDelegate next) =>
            {
                    //Write 1kb data in response
            });

Expected Behavior

Memory will not be much larger due to long concurrent requests

Steps To Reproduce

No response

Exceptions (if any)

No response

.NET Version

7.0.100

Anything else?

.NET SDK:
Version: 7.0.100
Commit: e12b7af219

运行时环境:
OS Name: Windows
OS Version: 10.0.19045
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\7.0.100\

Host:
Version: 7.0.0
Architecture: x64
Commit: d099f07

.NET SDKs installed:
3.1.426 [C:\Program Files\dotnet\sdk]
5.0.303 [C:\Program Files\dotnet\sdk]
5.0.403 [C:\Program Files\dotnet\sdk]
5.0.408 [C:\Program Files\dotnet\sdk]
6.0.112 [C:\Program Files\dotnet\sdk]
6.0.307 [C:\Program Files\dotnet\sdk]
7.0.100 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
Microsoft.AspNetCore.All 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.19 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.31 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.0-preview.4.22251.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.19 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.30 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.31 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.0-preview.4.22229.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.19 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 3.1.30 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 3.1.31 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 7.0.0-preview.4.22229.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 7.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
arm64 [C:\Program Files\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\arm64\InstallLocation]
x86 [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
Not set

global.json file:
Not found

Learn more:
https://aka.ms/dotnet/info

Download .NET:
https://aka.ms/dotnet/download

[adityamandaleeka]

@Cricle Are you seeing the memory usage go down eventually, or does it stay at 1G on .NET 7?

[Cricle]

@Cricle Are you seeing the memory usage go down eventually, or does it stay at 1G on .NET 7?

It stay above 1G

[davidfowl]

Are you running the application from Visual Studio?

[Cricle]

Are you running the application from Visual Studio?

No, I compiled with AOT or docker image run, both like that.

[davidfowl]

Can you reproduce this running outside of a container? The next step here would be to get a memory profile that compares the 2 applications.

cc @cshung @Maoni0 @mangod9

[Cricle]

I tried to reproduce that situation again today.
Compile with AOT, and wrk wrk -t8 -c1000 -d30s --latency http://192.168.2.86:8889, RPS halved, the memory will reach 275MB in a few seconds.
In docker run, after wrk, the static memory will be reduced to 188MB.

The computer has been restarted since the last test.

[Cricle]

Brief.zip
Here is the solution

https://hub.docker.com/r/hcricle/brief
Here is docker image

Here is compiled with AOT result
publish.zip

[mangod9]

Hey @Cricle, could you please clarify whether you notice the leak only when compiled as NativeAOT, or without it as well?

[Cricle]

Hey @Cricle, could you please clarify whether you notice the leak only when compiled as NativeAOT, or without it as well?

Only AOT

[mangod9]

Ok thanks for clarifying. Adding @MichalStrehovsky @VSadov in case they are aware of anything related to this.

[VSadov]

I am not aware of any known leaks specific to NativeAOT. From the description it looks like this is on Windows10.
I wonder if it is possible to create a memory dump of the app when it gets to 400-500 mb.

[davidfowl]

If the managed memory analysis tools work, we could take a look at what it might be.

[MichalStrehovsky]

@Cricle do you specify PublishAot anywhere? Looking at the dockerfile this is a r2r+trimmed+singlefile combo and not PublishAot like @mangod9 is suggesting.

Does the problem go away or persist if you remove any of publishsinglefile/publishtrimmed/publishreadytorun?

[Cricle]

@Cricle do you specify PublishAot anywhere? Looking at the dockerfile this is a r2r+trimmed+singlefile combo and not PublishAot like @mangod9 is suggesting.

Does the problem go away or persist if you remove any of publishsinglefile/publishtrimmed/publishreadytorun?

publish

Only PublishAOT but it will also.

csproj only

<PublishRelease>true</PublishRelease>

Command

dotnet publish -r win-x64 /p:PublishAot=true

Run debug in vs2022, no problem, stay 220mb

[adityamandaleeka]

@mangod9 Should this be transferred to the runtime repo?