Description
Testing GC Heap Counts with Containers
@Maoni0 asked me do to a test pass to validate the container limits doc we're updating. I thought it would be useful to share the results. The short version is the product did what I expected for the scenarios I tested.
Note: the cgroup values are only read on process startup. Changing these values afterwards will have no effect. Enabling a dynamic scenario could be considered in future.
Let's test!
The first set of tests are using my Apple M1 laptop. It's 8 cores but Docker Desktop limits itself to 4 cores by default (and I didn't change that).
I'll show all the steps the first time and then skip for the remaining examples.
dotnet-dump tool: https://docs.microsoft.com/dotnet/core/diagnostics/dotnet-dump
FYI: One of my reviewers mentioned that ps wasn't needed since dotnet-dump ps does the same thing but only lists .NET process, which actually makes the process easier.
1 Core
Note: that I'm using the linux-arm64 version of dotnet-dump. Link: https://aka.ms/dotnet-dump/linux-arm64
% docker run --rm --cpus 1 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
1b27f1306ab066735d4d49fe85bf2265673404883df788ccb013065833c898ac
% docker exec -it aspnetapp bash
/app# cd
~# apt update && apt install -y curl procps
~# curl -Lo dotnet-dump https://aka.ms/dotnet-dump/linux-arm64
~# chmod +x dotnet-dump
~# ps x
PID TTY STAT TIME COMMAND
1 ? Ssl 0:01 dotnet aspnetapp.dll
22 pts/0 Ss 0:00 bash
496 pts/0 R+ 0:00 ps x
~# ./dotnet-dump collect -p 1 -o dump.dmp
~# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 1
2 cores
% docker run --rm --cpus 2 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
/# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 2
2 cores + DOTNET_PROCESSOR_COUNT=3
% docker run --rm --cpus 2 -e DOTNET_PROCESSOR_COUNT=3 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
/# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 3
2 cores + DOTNET_PROCESSOR_COUNT=10
% docker run --rm --cpus 2 -e DOTNET_PROCESSOR_COUNT=10 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
/# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 4
Note: Docker desktop only gave me 4 cores on my laptop, as mentioned at the top. The DOTNET_PROCESSOR_COUNT=10 value is asking for more cores than the machine has.
3 cores (CPU affinity)
% docker run --rm --cpuset-cpus 0,2,3 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
/# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 3
Low memory; cores unconstrained
% docker run --rm -m 40mb -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 2
Ubconstrained
% docker run --rm -m -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 4
Unconstrained + DOTNET_GcHeapCount=2
% docker run --rm -e DOTNET_GCHeapCount=2 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 2
Big machine unconstrained
Note: I've switched to a big 64 core x64 machine in Azure, so am now using the linux-x64 version of dotnet-dump. Link: https://aka.ms/dotnet-dump/linux-x64.
~$ docker run --rm -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
~$ docker exec -it aspnetapp bash
~# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 64
Big machine -- memory constrained
~$ docker run --rm -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
~$ docker exec -it aspnetapp bash
~# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 12