Description
openedon Sep 22, 2023
We have several asp.net core service which are hosted in kubernetes.
After moving to NET7 we have experienced unexpected memory usage.
Memory is growing until kubernetes got killed container because of memory limit exceeding. But it is seems that there is no managed memory leaks because of based on Prometheus metrics heap stayed stable, and there is no OutOfMemoryException, at the same time working set is constantly growing.
So we have two dumps.
The first !sos maddress -images command output shows:
| Memory Type | Count | Size | Size (bytes) |
|---|---|---|---|
| Image | 301 | 83,99mb | 88?065?024 |
| PAGE_EXECUTE_READ | 1?418 | 41,16mb | 43?163?648 |
| PAGE_EXECUTE_WRITECOPY | 1?317 | 37,27mb | 39?082?496 |
| Stack | 19 | 25,95mb | 27?213?312 |
| GCHeap | 6 | 20,45mb | 21?442?560 |
| ------------------------ | ---------------- | -------------- | ---------------- |
| [TOTAL] | 3?547 | 240,06mb | 251?718?656 |
The same command on the second dump (after 3 days):
| Memory Type | Count | Size | Size (bytes) |
|---|---|---|---|
| Image | 454 | 185,48mb | 194?494?464 |
| PAGE_EXECUTE_WRITECOPY | 1?272 | 46,59mb | 48?853?504 |
| PAGE_EXECUTE_READ | 1?456 | 41,62mb | 43?642?880 |
| GCHeap | 5 | 29,97mb | 31?428?608 |
| Stack | 20 | 27,86mb | 29?216?256 |
| ------------------------ | ---------------- | -------------- | ---------------- |
| [TOTAL] | 3?706 | 363,52mb | 381?177?856 |
The detailed summary shows that a lot of memory used by doublemapper__deleted_ entry:
For the first dump:
| Image | Count | Size | Size (bytes) |
|---|---|---|---|
| doublemapper__deleted_ | 1 | 544,00kb | 557?056 |
And for the second dump:
| Image | Count | Size | Size (bytes) |
|---|---|---|---|
| doublemapper__deleted_ | 190 | 106,81mb | 111?996?928 |
Does it a memory leak? Is there any way to find out from where it comes from?
I will appreciate for any help, this issue becomes critical for us. Currently we are on pause with our migration to NET7 for other services.
Environment info:
Docker image: mcr.microsoft.com/dotnet/aspnet:7.0.10-alpine3.18
Memory limits: 512MB
Sdk: "Microsoft.NET.Sdk.Worker" (seems it is mean Workstation gc mode)