My server randomly hangs, with dumpexceptions showing System.ExecutionEngineException

[luukvnes]

Hi,

My asp.net server randomly stops working, while the process is still active and no exception is thrown. All threads "pause" and nothing happens anymore. As far as I can tell using dotnet-dump there is no deadlock, the syncblk table shows every line with Monitorheld 0. A strange thing i found is that using dumpexceptions I can see a System.ExecutionEngineException, but this does not contain any information about why or where this happened. I am not really sure where to go next to find the issue. If anyone could help me with finding the problem that would be amazing. I can share the dump privately if that helps.

edit: I forgot to mention this, but it seems constrained to happening on my linux(ubuntu) developer computer. My windows laptop and two linux(ubuntu) servers don't seem to have the same problem (so far).

---

   1 System.Threading.Monitor.Wait(Object, Int32)
   1 System.Threading.ManualResetEventSlim.Wait(Int32, CancellationToken)
   1 System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, CancellationToken)
   1 System.Threading.Tasks.Task.InternalWaitCore(Int32, CancellationToken)
   1 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions)
   1 Program.<Main>(String[])


________________________________________________
~~~~ 143232
   1 Interop+Sys.Read(SafeHandle, Byte*, Int32)
   1 System.IO.FileSystemWatcher+RunningInstance.TryReadEvent(NotifyEvent ByRef)
   1 System.IO.FileSystemWatcher+RunningInstance.ProcessEvents()


________________________________________________
~~~~ 1432ce
   1 System.Threading.ManualResetEventSlim.Wait(Int32, CancellationToken)
   1 System.Threading.ManualResetEventSlim.Wait(TimeSpan)
   1 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.Heartbeat.TimerLoop()
   1 System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object)


________________________________________________
     ~~~~ 143234
        1 System.Threading.Monitor.Wait(Object, Int32)
        1 System.Threading.ManualResetEventSlim.Wait(Int32, CancellationToken)
        1 System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, CancellationToken)
        1 System.Threading.Tasks.Task.InternalWaitCore(Int32, CancellationToken)
        1 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions)
        1 Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost)
        1 Program+<<Main>$>d__0.MoveNext()
        1 System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object)
        --------------- **Some startup code, removed here for clarity (running the program without this code does not help)** --------------------
        1 System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32, Memory<Byte>, SocketFlags, SocketError)
        1 System.Net.Sockets.SocketAsyncEngine.Execute()
        1 System.Threading.ThreadPoolWorkQueue.Dispatch()
     ~~~~ 55b8b
        1 System.Threading.LowLevelLifoSemaphore.WaitNative(SafeWaitHandle, Int32)
        1 System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32)
        1 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean)
   2 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()


________________________________________________
~~~~ 143235
   1 System.Threading.WaitHandle.WaitOneNoCheck(Int32, Boolean, Object, WaitHandleWaitSourceMap)
   1 System.Threading.PortableThreadPool+GateThread.GateThreadStart()


________________________________________________
~~~~ 143268
   1 System.Net.Sockets.SocketAsyncEngine.EventLoop()


________________________________________________
~~~~ 14326a
   1 System.Threading.WaitHandle.WaitOneNoCheck(Int32, Boolean, Object, WaitHandleWaitSourceMap)
   1 System.Threading.TimerQueue.TimerThread()


________________________________________________
~~~~ 1432b0
   1 System.Threading.Thread.Sleep(Int32)
   1 Program+<>c.<<Main>$>b__0_12()
   
 dumpexceptions                                                                                                                                                                                                                                                                                       
      Address      MethodTable Message Name
 7ad546404a48     7b15639527f0 Microsoft.AspNetCore.Connections.ConnectionAbortedException
     Message: The Socket transport's send loop completed gracefully.
 7ad546496b38     7b15639527f0 Microsoft.AspNetCore.Connections.ConnectionAbortedException
     Message: The Socket transport's send loop completed gracefully.
 7ad5465b37a8     7b15639527f0 Microsoft.AspNetCore.Connections.ConnectionAbortedException
     Message: The Socket transport's send loop completed gracefully.
 7ad57c000088     7b155c1ce440 System.OutOfMemoryException
     Message: <null>
 7ad57c000100     7b155c1ce568 System.StackOverflowException
     Message: <null>
 7ad57c000178     7b155c1ce690 System.ExecutionEngineException
     Message: <null>
 7ad57c36b5d8     7b155e587a08 System.ObjectDisposedException
     Message: Cannot access a disposed object.
 7ad546c20d70     7b15639527f0 Microsoft.AspNetCore.Connections.ConnectionAbortedException
     Message: The Socket transport's send loop completed gracefully.
 7ad546d5dfc0     7b15639527f0 Microsoft.AspNetCore.Connections.ConnectionAbortedException
     Message: The Socket transport's send loop completed gracefully.
 7ad57e5d9280     7b155df682e0 System.ArgumentNullException
     Message: Value cannot be null.

   Total: 10 objects
   
 bottom part of dumpheap
 7b155e4e4ea0  2,545    386,840 System.Reflection.Emit.DynamicILGenerator
7b1561a850b0  1,546    494,720 System.Collections.Concurrent.ConcurrentDictionary<System.String, Microsoft.EntityFrameworkCore.Infrastructure.Annotation>+VolatileNode[]
7b155e4f4720  5,889    518,232 System.Reflection.RuntimeParameterInfo
7b155c1c8d38  7,710    731,324 System.Int32[]
7b155e4f1a40  7,456    775,424 System.Reflection.RuntimeMethodInfo
7b155c277d30  1,702    939,938 System.Char[]
7b155c172278  6,388  1,078,648 System.Object[]
7b155b234910 64,271  1,542,504 System.Object
7b155c1cbe30 44,794  6,382,110 System.String
7b155e3a9240 14,496  8,133,504 System.Byte[]
5633de175bd0  3,884 13,070,488 Free
Total 386,593 objects, 48,672,038 bytes

Fragmented blocks larger than 0.5 MB:
        Address           Size      Followed By
   7ad54be45858        800,048     7ad54bf08d88 System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder<Npgsql.IBackendMessage>+StateMachineBox<Npgsql.Internal.NpgsqlConnector+<ReadMessageLong>d__235>
   7ad57fdad190        935,600     7ad57fe91840 System.Buffers.SharedArrayPoolPartitions

[mangod9]

Are you able to share a dump of this issue? Is it consistently reproing?

[luukvnes]

It is reproing consistently in that it always happens at some point. However after how long it happens is incosistent.

This is a link to the dump file

https://we.tl/t-ytAolSuhUR

It expires in three days. If you haven't gotten it by then let me know and i can create a new link.

[mangod9]

Looking at the dump I am noticing a few threads with this stack, are you certain the JobScheduler has work and isn't deadlocked for some reason?

00007A0F2E7FB550 00007a5171ec3a60 libcoreclr!JIT_RareDisableHelper + 0x17 [/__w/1/s/src/coreclr/vm/amd64/unixasmhelpers.S:134], calling libcoreclr!JIT_RareDisableHelperWorker [/__w/1/s/src/coreclr/vm/jithelpers.cpp:3608]
00007A0F2E7FB590 00007a50fb6cbf03 (MethodDesc 00007a50f5883370 + 0x173 Interop+Kernel32.CreateEventEx(IntPtr, System.String, UInt32, UInt32)), calling 00007a5171d662a0 (stub for System.Runtime.InteropServices.Marshal.SetLastPInvokeError(Int32))
00007A0F2E7FB5A0 00007a5171d1fea6 libcoreclr!WaitHandleNative::CorWaitOneNative + 0x96 [/__w/1/s/src/coreclr/vm/comwaithandle.cpp:24], calling libcoreclr!LazyMachStateCaptureState [/__w/1/s/src/coreclr/vm/amd64/getstate.S:28]
00007A0F2E7FB5F0 00007a50fb56c960 (MethodDesc 00007a50f7fa64c0 + 0xc0 System.Threading.WaitHandle.WaitOneNoCheck(Int32, Boolean, System.Object, WaitHandleWaitSourceMap)), calling 00007a5171d1fe10 (stub for System.Threading.WaitHandle.WaitOneCore(IntPtr, Int32, Boolean))
00007A0F2E7FB640 00007a50fbc61b46 (MethodDesc 00007a50fae429c8 + 0x1e6 Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan)), calling (MethodDesc 00007a50f7fa64c0 + 0 System.Threading.WaitHandle.WaitOneNoCheck(Int32, Boolean, System.Object, WaitHandleWaitSourceMap))
00007A0F2E7FB6D0 00007a50fc2644cf (MethodDesc 00007a50fad544e0 + 0xcf Hangfire.Server.RecurringJobScheduler.Execute(Hangfire.Server.BackgroundProcessContext)), calling (MethodDesc 00007a50fae429c8 + 0 Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan))
00007A0F2E7FB730 00007a50fcf96e60 (MethodDesc 00007a50fad555c0 + 0x80 Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object))
00007A0F2E7FB760 00007a50fa31e31c (MethodDesc 00007a50fad555c0 + 0x44c Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object)), calling libcoreclr!JIT_Patchpoint [/__w/1/s/src/coreclr/vm/jithelpers.cpp:3833]
00007A0F2E7FB8C0 00007a50fa3184cf (MethodDesc 00007a50fad5c8d8 + 0x1df Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object))
00007A0F2E7FBA20 00007a50fa317e75 (MethodDesc 00007a50fad5d0d8 + 0x55 Hangfire.Processing.BackgroundDispatcher.DispatchLoop())
00007A0F2E7FBAA0 00007a50f2c910a6 (MethodDesc 00007a50f60bb940 + 0x96 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))

[luukvnes]

I also thought at first that it would be due to hangfire. But even when I remove hangfire completely from my project the issue remains. I created another dump with that situation. It again expires in three days.

https://we.tl/t-MuSL03Kuho