How to debug memory leak that is not clear on dotnet-dump?

[Leonardo-Ferreira]

I have an asp.net 7.0.9 api running on AKS and no one is using it... and yet, every minute, application insights reports that the memory usage grew a bit... it grows until the pod crashes and it starts again... take a look:

I got a dump in a moment where the app insights was reporting that the app was using barely above 300MB and when I opened it on VS I got this:

If you look closely, the first object is smaller than 1.5MB and the total sum is 53k objects and 4.8MB...

I think its clear that I won't be able to track my leak here, so what should I do now?

[hoyosjs]

@Leonardo-Ferreira In the insights tab or using perfview or WinDBG + SOS you should check what your fragmentation looks like. Also, what does checking the "show dead objects" checkbox show?

[Leonardo-Ferreira]

Dead object show a bit more data here... but not enough to account for the 300MB...

The insights tab show a bit of waste on duplicated strings (26k objects totaling 603Kb) and some Sparse Arrays (24k objects totaling 1.36MB)

im trying to install the SOS on my pod but I must say im getting my ass kicked...

Tagging subscribers to this area: @tommcdon
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I have an asp.net 7.0.9 api running on AKS and no one is using it... and yet, every minute, application insights reports that the memory usage grew a bit... it grows until the pod crashes and it starts again... take a look:

I got a dump in a moment where the app insights was reporting that the app was using barely above 300MB and when I opened it on VS I got this:

If you look closely, the first object is smaller than 1.5MB and the total sum is 53k objects and 4.8MB...

I think its clear that I won't be able to track my leak here, so what should I do now?

Author:	Leonardo-Ferreira
Assignees:	-
Labels:	area-Diagnostics-coreclr, untriaged, needs-area-label
Milestone:	-

[hoyosjs]

You can open that dump in WinDBG on windows - no need to install within the container. You may need a copy of the binaries/pdbs for some metadata/commands. Alternatively dotnet-dump also works.

[tommcdon]

Moving this issue to dotnet-diagnostics as this is a tooling question

[Leonardo-Ferreira]

@hoyosjs I have been unable to reproduce this issue in windows, or even on docker... and the dump I've taken in the AKS pods, I did using dotnet-dump...

[tommcdon]

@Leonardo-Ferreira we support Linux x64 cross-plat dumps in Visual Studio and Windbg, so you can copy the Linux dump to Windows and open it in VS/Windbg, for example.

[Leonardo-Ferreira]

im really sorry @tommcdon, I did opened it on VS and posted reference screenshots, im just lost here kinda of going a bit crazy...

when I try to use windbg, I get an error when using commands such as !heap -? saying that no heap was exported... and this dump was collected using the command ./dotnet-dump collect -p 1 which was supposed to be a full dump, right? and now that I mentioned it, VS only shows "Debug Managed Memory" option...

On a sidebar question, if the memory leaked, it is not supposed to be trackable via managed memory, right?

[Leonardo-Ferreira]

using dotnet counters monitor I was able to get this:

the working set for the process is 363MB but I still have no clue "where" that memory is going...