The app crashes under .NET7 in Docker with "Fatal error. The RW block to unmap was not found"

[SergeiPavlov]

Description

When we switch to .NET7 from .NET6 we often see "Fatal error. The RW block to unmap was not found"

There are 2 places in dotnet code with this message:
https://github.com/dotnet/runtime/blob/main/src/coreclr/utilcode/executableallocator.cpp#L280
https://github.com/dotnet/runtime/blob/main/src/coreclr/utilcode/executableallocator.cpp#L845

Under .NET6 never seen such an error.
DOTNET_EnableWriteXorExecute=0 setting helps to avoid it, but then OutOfMemory does appear, even when there is enough free memory for Heap and unmanaged needs.

Reproduction Steps

.NET7 ASP NET app under Docker-in-Docker environment

Expected behavior

No crash

Actual behavior

Stack trace may be various. Example:

Fatal error. The RW block to unmap was not found
   at System.Reactive.Concurrency.DefaultScheduler.Schedule[[System.ValueTuple`2[[System.Reactive.Concurrency.LocalScheduler, System.Reactive, Version=5.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263],[System.Reactive.Disposables.SingleAssignmentDisposable, System.Reactive, Version=5.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263]], System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.ValueTuple`2<System.Reactive.Concurrency.LocalScheduler,System.Reactive.Disposables.SingleAssignmentDisposable>, System.Func`3<System.Reactive.Concurrency.IScheduler,System.ValueTuple`2<System.Reactive.Concurrency.LocalScheduler,System.Reactive.Disposables.SingleAssignmentDisposable>,System.IDisposable>)
   at System.Reactive.Concurrency.DefaultScheduler.Schedule[[System.ValueTuple`2[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.ValueTuple`2<System.__Canon,System.__Canon>, System.TimeSpan, System.Func`3<System.Reactive.Concurrency.IScheduler,System.ValueTuple`2<System.__Canon,System.__Canon>,System.IDisposable>)
   at System.Reactive.Concurrency.LocalScheduler.ScheduleShortTermWork(WorkItem)
   at System.Reactive.Concurrency.LocalScheduler.EvaluateLongTermQueue()
   at System.Reactive.Concurrency.LocalScheduler.SystemClockChanged(System.Object, System.Reactive.PlatformServices.SystemClockChangedEventArgs)
   at System.Reactive.PlatformServices.SystemClock.OnSystemClockChanged(System.Object, System.Reactive.PlatformServices.SystemClockChangedEventArgs)
   at System.Reactive.PlatformServices.PeriodicTimerSystemClockMonitor.TimeChanged()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.TimerQueueTimer.Fire(Boolean)
   at System.Threading.TimerQueue.FireNextTimers()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
[createdump] Problem suspending threads: ptrace(ATTACH, 6) FAILED Operation not permitted (1)
[createdump] Failure took 0ms

Regression?

On .NET6 it does work.

Known Workarounds

DOTNET_EnableWriteXorExecute=0 but it induces OutOfMemory exception more often

Configuration

Based on image mcr.microsoft.com/dotnet/aspnet:7.0.2-bullseye-slim-amd64
We have limited .NET memory consumption by settings:
Docker-compose's

    mem_limit: 39g
    mem_reservation: 36g

Env var:
DOTNET_GCHeapHardLimit: "3C0000000"

No response

Other information

No response

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[jkotas]

cc @janvorli

[janvorli]

@SergeiPavlov would you be able to create a simple repro that you could share? We have not seen this issue ever before, so it seems there is something special going on in your case. I am referring to the "Fatal error. The RW block to unmap was not found" issue.

[SergeiPavlov]

@SergeiPavlov would you be able to create a simple repro that you could share? We have not seen this issue ever before, so it seems there is something special going on in your case. I am referring to the "Fatal error. The RW block to unmap was not found" issue.

Unfortunately I cannot reproduce it on simple example. It crashes on huge app unit-testing process at relatively random time.
The first suspicion was "a deficite of RAM", but logging of GCMemoryInfo shows no signs of memory stress condition: HeapSizeBytes & Process.WorkingSet64 < 8G, while 36GB is reserved and ~16GB is allowed to .NET (DOTNET_GCHeapHardLimit)

[janvorli]

@SergeiPavlov ok, in that case, would it be possible to share a core dump at the failure? Or if that would not work for you, try to use a private build of libcoreclr.so with some extra logging that I would add?

[SergeiPavlov]

try to use a private build of libcoreclr.so with some extra logging that I would add?

Sorry, I cannot provide coredump.
I'm ready to run tests with your libcoreclr.so .

[janvorli]

Great, let me prepare it then. I will assume that you'll use the mcr.microsoft.com/dotnet/aspnet:7.0.2-bullseye-slim-amd64 image for the testing, as I need to build the libcoreclr.so from a specific state of the source tree that matches what's in the image.

[janvorli]

@SergeiPavlov you can get the testing libcoreclr.so here: https://1drv.ms/u/s!AkLV4wRkyHYhymkJZQ-qrTEFcxpI?e=GrlQyt
It is a drop-in replacement for the one in mcr.microsoft.com/dotnet/aspnet:7.0.2-bullseye-slim-amd64. You'll need to copy it over the one that's installed there.

Then you'll also need to set these env variables before running your app to enable the logging:

COMPlus_StressLogSize=2000000
COMPlus_TotalStressLogSize=1
COMPlus_StressLog=1
COMPlus_LogFacility=80
COMPlus_LogLevel=7
COMPlus_StressLogFilename=/path/to/log/file

The /path/to/log/file is a path to where the log file will be written. It will have size determined by the COMPlus_TotalStressLogSize (in GB). For now, it is set to 1GB, we'll see if that's enough or not.

After the process crashes, I'd need you to gzip the log file (it should compress well) and share it with me. I will then run a StressLogAnalyzer tool that's built in the runtime repo to view the events around the executable memory mapping. I have added events that log all mappings, unmappings and reservations. That should show us where we try to unmap something that was not mapped or already unmapped before.

[SergeiPavlov]

Thank you. I will try it.

[SergeiPavlov]

I have collected the log.
Extended TotalStressLogSize to 22GB.
The error is flaky. I see Segmentation fault more often than The RW block to unmap was not found.
So I think it is a side effect of memory corruption by some unmanaged or unsafe code.

[janvorli]

@SergeiPavlov it seems that the log was not produced using the testing libcoreclr.so or the COMPlus_LogFacility=80 was not set. It doesn't contain any of the logging messages I've added. How did you replace the libcoreclr.so? If your application is not self contained, then you need to overwrite the /usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.2/libcoreclr.so. If it is self contained, you'll need to overwrite the libcoreclr.so present in the subtree of your application.

[SergeiPavlov]

The app is not self contained.
Oh, my mistake. I see the test's folder has its own libcoreclr.so. Actually it is self-contained

replaced by following Dockerfile instruction:

rm /usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.2/libcoreclr.so && wget -O /usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.2/libcoreclr.so http://`<filestorage.url>`/libcoreclr.so && echo libcoreclr.so has been replaced

I've checked that libcoreclr.so is exactly yours:

root@vm:/app# docker exec -it 22bbcf3d3b14 /bin/bash
root@22bbcf3d3b14:/app# find /usr -name libcoreclr.so
/usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.2/libcoreclr.so
root@22bbcf3d3b14:/app# ls -la /usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.2/libcoreclr.so
-rw-r--r-- 1 root root 6983112 Jan 17 01:01 /usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.2/libcoreclr.so
root@22bbcf3d3b14:/app# env | grep COMPlus
COMPlus_LogLevel=7
COMPlus_StressLogFilename=/tmp/stress.log
COMPlus_LogFacility=80
COMPlus_TotalStressLogSize=8
COMPlus_StressLogSize=20000000
COMPlus_StressLog=1
root@22bbcf3d3b14:/app#