GC hangs indefinitely on LongRunning

[cleardarkz]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

Sometimes when running a background worker for a lengthy period of time, it appears that the GC tries to run a collection round, suspends all threads but gets interlocked while trying to free resources

We have experienced this behavior on long running processes, usually a couple of hours or more. I will add that the codebase is using GetAwaiter.Result() extensively, however, it appears that even in this case, there should be some sort of a fallback to when the GC fails to run

I have attached a dump diagram of the latest occurrence. I will be able to provide more logs/dumps as required.

Attempting to reproduce the issue with a sample repo succeeded in behavior but the memory dumps do not look the same, therefor the sample project is not attached to the report

Expected Behavior

Resources allocation exception, process exception, any sort of an error which will resume the .NET internals

Steps To Reproduce

No response

Exceptions (if any)

No exceptions, the process hangs until manually freed (Forceful shutdown)

.NET Version

8.0.409

Anything else?

[martincostello]

Looks like this issue should be transferred to dotnet/runtime as it doesn't appear to be specific to ASP.NET Core.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

[mangod9]

@cleardarkz, can you please provide a repro for this issue or a dump when its deadlocked?

[cleardarkz]

@cleardarkz, can you please provide a repro for this issue or a dump when its deadlocked?

Absolutely,

Here's a Google Drive link:
https://drive.google.com/file/d/1_7KcDgkSuqIA9MFpe_kqkxXE3bbdL9b7/view?usp=sharing

[mangod9]

Is this a NativeAOT app? I notice that ThreadSuspension appears to be blocked by this reverse pInvoke stack. @VSadov have you seen anything like this before?

 # Call Site
00 ntdll!ZwWaitForSingleObject
01 KERNELBASE!WaitForSingleObjectEx
02 Cymulate_Agent_Service!Thread::RareDisablePreemptiveGC
03 Cymulate_Agent_Service!JIT_ReversePInvokeEnterRare2
04 Cymulate_Agent_Service!JIT_ReversePInvokeEnterTrackTransitions
05 0x0
06 user32!UserCallWinProcCheckWow
07 user32!DispatchClientMessage
08 user32!__fnINDEVICECHANGE
09 ntdll!KiUserCallbackDispatcherContinue
0a win32u!ZwUserGetMessage
0b user32!GetMessageW
0c 0x0
0d 0x0
0e Cymulate_Agent_Service!CallDescrWorkerInternal
0f Cymulate_Agent_Service!CallDescrWorkerWithHandler
10 Cymulate_Agent_Service!DispatchCallSimple
11 Cymulate_Agent_Service!ThreadNative::KickOffThread_Worker
12 Cymulate_Agent_Service!ManagedThreadBase_DispatchInner
13 Cymulate_Agent_Service!ManagedThreadBase_DispatchMiddle
14 Cymulate_Agent_Service!ManagedThreadBase_DispatchOuter
15 Cymulate_Agent_Service!ManagedThreadBase_FullTransition
16 Cymulate_Agent_Service!ManagedThreadBase::KickOff
17 Cymulate_Agent_Service!ThreadNative::KickOffThread
18 kernel32!BaseThreadInitThunk
19 ntdll!RtlUserThreadStart

[VSadov]

Is this a NativeAOT app? I notice that ThreadSuspension appears to be blocked by this reverse pInvoke stack. @VSadov have you seen anything like this before?

This is a normal stack for a thread that is trying to enter managed code while EE is suspended, most likely for GC. In such case the thread will block and wait for EE to resume.

[VSadov]

From the screenshot it looks like

• it is CoreCLR, not NativeAot (because PulseAll is in native code)
• one thread tries to initiate GC, thus it starts suspending all managed threads
• almost all threads are suspended, but not all.
• the suspending thread is waiting (with timeout) for more progress. If the wait times out it will try checking if threads are all stopped or can be forced to stop via hijacking, interrupts etc..
  Ultimately it is possible for a thread to be in a state where it cannot be suspended. Such states should be transient and should not take long.
• one thread is running ObjectNative::PulseAll. If it runs in COOP mode it would be in a "can't suspend" state and would prevent EE suspend to complete.

If the PulseAll thread is somehow stuck, EE cannot finish suspending and GC cannot start.

I am not sure how PulseAll can get stuck though - it is just going through a linked list of waiters and sets every waiter's event.

The waiter list would be limited by the number of threads in the program, so should not be long. Perhaps it gets corrupted somehow and became circular?
Or maybe the Set on the event gets stuck - not sure if that can happen.

[cymulateagentteam]

@VSadov I have reproduced the issue and debugged it live.

The singly linked list had somehow turned into a circular list, which is definitely an issue.
The memory hasn't been corrupted, the link->next value pointed to the same node. See the following:

runtime/src/coreclr/vm/syncblk.cpp

Line 240 in 7e7e195

psb->m_Link.m_pNext = pLink->m_pNext;

causing the following PulseAll to effectively deadlock the GC collection procedure

runtime/src/coreclr/vm/syncblk.cpp

Line 2877 in 7e7e195

while ((pWaitEventLink = ThreadQueue::DequeueThread(this)) != NULL)

I have managed to unclog the deadlock in runtime by manually resetting the ->m_pNext value to NULL, restoring the SList expected behavior

I believe the root cause might be manipulation of the SList from code that's unprotected by a critical section

There could be an easy mitigation for this issue, with the following before L240:

if ( psb->m_Link.m_pNext == pLink->m_pNext )
psb->m_Link.m_pNext = NULL;