Net 7 Kestrel windows service hangs after a period of time

[sccrgoalie1]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

We just update our projects to .NET 7 from .NET 5 and are experiencing hangs where Kestrel quits responding. It works for a day or so and then the windows service needs to be restarted. This was not an issue on .NET5. I gathered a memory dump to see if I can figure out what is causing it to hang. Here are some screenshots. This was gathered using ProcDump.

This API is installed at many unique locations and several have reported the same issue that they need to restart the service daily. For some locations, we use Azure relay. So far, the locations that go through the relay have not had any issues.

Expected Behavior

The API should respond while running

Steps To Reproduce

Let our project run for a day

Exceptions (if any)

No response

.NET Version

7.0.100

Anything else?

No response

[danmoseley]

Just curious does it repro on 6.0?

[sccrgoalie1]

Just curious does it repro on 6.0?

Not sure, we skipped over 6.0 and went straight to 7.0. We might be able to backtrack if that will help in diagnosing.

[davidfowl]

There was another issue like this with similar symptoms (hanging after a day or something). Can you look at parallel stacks?

[sccrgoalie1]

There was another issue like this with similar symptoms (hanging after a day or something).

Yeah, it was this one dotnet/aspnetcore#45215. I reviewed that one and I don't think a solution was found, just not enough info to diagnose.

[davidfowl]

@sccrgoalie1 Can you run through all of the same steps?

[sccrgoalie1]

@davidfowl Absolutely, we ran through that issue extensively before we submitted our own. Was there something specific that would be helpful? I have the memory dump. All the requests are http not https. So far, it seems that the issue occurs after the API sits idle for a period of time. Our users work banker's hours, so it works all day long, but when they come back the next morning Kestrel is no longer responding to any requests until we restart the windows service it's running in.

[adityamandaleeka]

@sccrgoalie1 Is it possible for you to share the dump with us (we'd understand if you cannot)?

[adityamandaleeka]

Can you share more info about what exactly happens on the client and server when this happens?

For instance, does the client get a response back? Do the server logs show anything interesting? Perhaps you may get some interesting output if you enable more detailed logging on the server. See here for more info on configuring logging: https://learn.microsoft.com/en-us/aspnet/core/fundamentals/logging/?view=aspnetcore-7.0#configure-logging

[sccrgoalie1]

@sccrgoalie1 Is it possible for you to share the dump with us (we'd understand if you cannot)?

@adityamandaleeka Sure, here is the memory dump.

Memory Dump

[sccrgoalie1]

Can you share more info about what exactly happens on the client and server when this happens?

For instance, does the client get a response back? Do the server logs show anything interesting? Perhaps you may get some interesting output if you enable more detailed logging on the server. See here for more info on configuring logging: https://learn.microsoft.com/en-us/aspnet/core/fundamentals/logging/?view=aspnetcore-7.0#configure-logging

The client does not get a response back, the requests just hang. Unfortunately, we haven't found anything in the logs or event viewer yet. We'll try increasing the detailed logging to see if we can get anything better. The windows service on the server is still running but just not responding.

[adityamandaleeka]

@sccrgoalie1 Thanks for sharing the dump. I looked through it and didn't spot any immediate issues that jumped out. Just curious, did taking the dump cause the process to "unhang" or was it still in the stuck/unresponsive state?

[adityamandaleeka]

As a side note (maybe not related to the problem you're hitting), I see a bunch of threads doing things related to performance counters, which are running code from aspnet_perf.dll. Do you know what's causing that?

AFAICT aspnet_perf is an old ASP.NET framework component so I'm wondering what it's doing in the context of a ASP.NET Core app.

[sccrgoalie1]

@sccrgoalie1 Thanks for sharing the dump. I looked through it and didn't spot any immediate issues that jumped out. Just curious, did taking the dump cause the process to "unhang" or was it still in the stuck/unresponsive state?

No, taking the dump did not unstick it. At one of the sites having to restart daily we converted them to use Microsoft.Azure.Relay.AspNetCore and since then they have been running for days without any issues. I'm not sure at what level that package intertwines with Kestrel, but it seems to prevent the hang we are experiencing.

[sccrgoalie1]

As a side note (maybe not related to the problem you're hitting), I see a bunch of threads doing things related to performance counters, which are running code from aspnet_perf.dll. Do you know what's causing that?

AFAICT aspnet_perf is an old ASP.NET framework component so I'm wondering what it's doing in the context of a ASP.NET Core app.

The only thing I can think of that might be doing that is Application Insights but we are using the NetCore version (Microsoft.ApplicationInsights.AspNetCore)

[heikkilamarko]

We have several .NET 7.0.2 applications running on Windows Server 2019. We run the apps as HashiCorp Nomad Jobs.

We noticed that HTTP requests to an application start to fail if we open the Windows "Resource Monitor" application and select the dotnet process running our app from the list (toggle checkbox on).
After selecting the application, it takes some time before HTTP requests, such as health checks start to fail (hang). To make the app work again, restart is required.

You can also select more than one apps, and all the selected apps start to fail. Apps that are not selected work without problems.

The problem reproduces consistently.