Access Violation in jithelpers with Dynamic PGO enabled

[arekpalinski]

We have a scenario in which we are getting an access violation exception during a particular state of our RavenDB cluster when Dynamic PGO is enabled.

The setup is 3 servers, 2 of them are down and a client app is constantly trying to connect via TCP to the running server.
The connections are established but then some of the connections are dropped because they were supposed to by handled by a different node (which is down).

We have a way to reproduce it. The key seems to be that it is happening only with Dynamic PGO enabled. Disabling it makes that no crash is experienced.

We have a full dump of the process when the crash happened, available here:
https://drive.google.com/file/d/1EC0Gwz_ljWuCdJX2WaKDxfdGMpoINnba/view?usp=sharing

We're running using:

Architecture: x64
RuntimeFrameworkVersion: 7.0.7
TieredPGO: True

Exception

(72b4.5ee4): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
coreclr!JIT_ClassInitDynamicClass_Helper+0x7c:
00007ffd`cd4512e8 488b0cc8        mov     rcx,qword ptr [rax+rcx*8] ds:00008005`7003e060=????????????????

Stacktrace

0:093> !clrstack
OS Thread Id: 0x5ee4 (93)
        Child SP               IP Call Site
000000FF8C2BE258 00007ffdcd4512e8 [HelperMethodFrame: 000000ff8c2be258] 
000000FF8C2BE390 00007ffd70f4efc7 Raven.Server.RavenServer.CreateSubscriptionConnection(Raven.Server.ServerWide.ServerStore, DatabaseSearchResult, Raven.Server.Documents.TcpHandlers.TcpConnectionOptions, MemoryBuffer)
000000FF8C2BE430 00007ffd70f4e31e Raven.Server.RavenServer+d__99.MoveNext() [D:\workspace\ravendb-6.0\src\Raven.Server\RavenServer.cs @ 2463]
000000FF8C2BE520 00007ffd70f4dbff System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Raven.Server.RavenServer+d__99, Raven.Server]](d__99 ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
000000FF8C2BE580 00007ffd709c38b0 Raven.Server.RavenServer+d__79.MoveNext() [D:\workspace\ravendb-6.0\src\Raven.Server\RavenServer.cs @ 2074]
000000FF8C2BE690 00007ffd709c368f System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Raven.Server.RavenServer+d__79, Raven.Server]](d__79 ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
000000FF8C2BE6F0 00007ffd709c35e2 Raven.Server.RavenServer.DispatchTcpConnection(Raven.Client.ServerWide.Tcp.TcpConnectionHeaderMessage, Raven.Server.Documents.TcpHandlers.TcpConnectionOptions, MemoryBuffer, System.Security.Cryptography.X509Certificates.X509Certificate2)
000000FF8C2BE760 00007ffd7095af5c Raven.Server.RavenServer+c__DisplayClass78_0+<b__0>d.MoveNext() [D:\workspace\ravendb-6.0\src\Raven.Server\RavenServer.cs @ 2006]
000000FF8C2BE930 00007ffd7039a166 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183]
000000FF8C2BE9A0 00007ffd7096d919 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib],[Raven.Server.RavenServer+c__DisplayClass78_0+<b__0>d, Raven.Server]].MoveNext(System.Threading.Thread) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 324]
000000FF8C2BEA10 00007ffd703f4f07 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs @ 795]
000000FF8C2BEA80 00007ffd703f3db7 System.Threading.Tasks.Task.RunContinuations(System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 3411]
000000FF8C2BEB20 00007ffd704da7d9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.__Canon>, System.__Canon) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 446]
000000FF8C2BEB70 00007ffd7096f981 Raven.Server.RavenServer+d__80.MoveNext() [D:\workspace\ravendb-6.0\src\Raven.Server\RavenServer.cs @ 2208]
000000FF8C2BEF90 00007ffd7090b396 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Raven.Server.RavenServer+d__80, Raven.Server]].ExecutionContextCallback(System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 286]
000000FF8C2BEFC0 00007ffd7039a166 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183]
000000FF8C2BF030 00007ffd7090b1d7 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Raven.Server.RavenServer+d__80, Raven.Server]].MoveNext(System.Threading.Thread) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 324]
000000FF8C2BF0C0 00007ffd70926f8a System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Raven.Server.RavenServer+d__80, Raven.Server]].MoveNext() [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 302]
000000FF8C2BF0F0 00007ffd703f4f07 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs @ 795]
000000FF8C2BF160 00007ffd703f3db7 System.Threading.Tasks.Task.RunContinuations(System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 3411]
000000FF8C2BF200 00007ffd704da7d9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.__Canon>, System.__Canon) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 446]
000000FF8C2BF250 00007ffd709329c3 Sparrow.Json.JsonOperationContext+d__65.MoveNext() [D:\workspace\ravendb-6.0\src\Sparrow\Json\JsonOperationContext.cs @ 835]
000000FF8C2BF6F0 00007ffd7039a166 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183]
000000FF8C2BF760 00007ffd7099ee17 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Sparrow.Json.JsonOperationContext+d__65, Sparrow]].MoveNext(System.Threading.Thread) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 324]
000000FF8C2BF7D0 00007ffd7091f137 System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.InvokeContinuation(System.Action`1<System.Object>, System.Object, Boolean, Boolean) [/_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs @ 1321]
000000FF8C2BF840 00007ffd7091effc System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.OnCompleted(System.Net.Sockets.SocketAsyncEventArgs) [/_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs @ 1004]
000000FF8C2BF8B0 00007ffd709063f0 System.Net.Sockets.SocketAsyncEventArgs+c.<.cctor>b__176_0(UInt32, UInt32, System.Threading.NativeOverlapped*) [/_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Windows.cs @ 1250]
000000FF8C2BF910 00007ffd704daf5f System.Threading.ThreadPoolTypedWorkItemQueue`2[[System.Threading.PortableThreadPool+IOCompletionPoller+Event, System.Private.CoreLib],[System.Threading.PortableThreadPool+IOCompletionPoller+Callback, System.Private.CoreLib]].System.Threading.IThreadPoolWorkItem.Execute() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs @ 1134]
000000FF8C2BFA00 00007ffd703f47fa System.Threading.ThreadPoolWorkQueue.Dispatch() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs @ 924]
000000FF8C2BFA80 00007ffdcc09f2a7 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 77]
000000FF8C2BFDE0 00007ffdcd48fb73 [DebuggerU2MCatchHandlerFrame: 000000ff8c2bfde0] 

0:093> !dumpstack
OS Thread Id: 0x5ee4 (93)
Current frame: coreclr!JIT_ClassInitDynamicClass_Helper + 0x7c [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1482]
Child-SP         RetAddr          Caller, Callee
000000FF8C2BE260 00007ffd70f505f8 (MethodDesc 00007ffd708ecfe0 + 0x78 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Raven.Server.Documents.Subscriptions.SubscriptionBinder`3+<Run>d__6[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]], Raven.Server]](<Run>d__6<System.__Canon,System.__Canon,System.__Canon> ByRef))
000000FF8C2BE2D0 00007ffd70f504dc (MethodDesc 00007ffd708eae70 + 0xac Raven.Server.Documents.Subscriptions.SubscriptionBinder`3[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].Run(Raven.Server.Documents.TcpHandlers.TcpConnectionOptions, System.IDisposable)), calling (MethodDesc 00007ffd708ecd20 + 0 System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Raven.Server.Documents.Subscriptions.SubscriptionBinder`3+<Run>d__6[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]], Raven.Server]](<Run>d__6<System.__Canon,System.__Canon,System.__Canon> ByRef))
000000FF8C2BE318 00007ffdcd4512c9 coreclr!JIT_ClassInitDynamicClass_Helper + 0x5d [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1477], calling coreclr!LazyMachStateCaptureState
000000FF8C2BE350 00007ffdcd45126a coreclr!JIT_ClassInitDynamicClass + 0x2a [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1503], calling coreclr!JIT_ClassInitDynamicClass_Helper [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1473]
000000FF8C2BE380 00007ffd70f4efc7 (MethodDesc 00007ffd6dd70550 + 0x3e7 Raven.Server.RavenServer.CreateSubscriptionConnection(Raven.Server.ServerWide.ServerStore, DatabaseSearchResult, Raven.Server.Documents.TcpHandlers.TcpConnectionOptions, MemoryBuffer)), calling coreclr!JIT_ClassInitDynamicClass [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1491]
000000FF8C2BE420 00007ffd70f4e31e (MethodDesc 00007ffd6ecce520 + 0x64e Raven.Server.RavenServer+<DispatchDatabaseTcpConnection>d__99.MoveNext()), calling (MethodDesc 00007ffd6dd70550 + 0 Raven.Server.RavenServer.CreateSubscriptionConnection(Raven.Server.ServerWide.ServerStore, DatabaseSearchResult, Raven.Server.Documents.TcpHandlers.TcpConnectionOptions, MemoryBuffer))
000000FF8C2BE510 00007ffd70f4dbff (MethodDesc 00007ffd708e9c30 + 0x4f System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Raven.Server.RavenServer+<DispatchDatabaseTcpConnection>d__99, Raven.Server]](<DispatchDatabaseTcpConnection>d__99 ByRef)), calling (MethodDesc 00007ffd6ecce520 + 0 Raven.Server.RavenServer+<DispatchDatabaseTcpConnection>d__99.MoveNext())
000000FF8C2BE520 00007ffd704e845f (MethodDesc 00007ffd6f577bf8 + 0x4f System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.Release()), calling 00007ffdcd442610 (stub for System.Threading.Interlocked.CompareExchange(System.Object ByRef, System.Object, System.Object))
000000FF8C2BE570 00007ffd709c38b0 (MethodDesc 00007ffd6ecd0180 + 0x150 Raven.Server.RavenServer+<DispatchTcpConnection>d__79.MoveNext()), calling (MethodDesc 00007ffd708e9c30 + 0 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Raven.Server.RavenServer+<DispatchDatabaseTcpConnection>d__99, Raven.Server]](<DispatchDatabaseTcpConnection>d__99 ByRef))
000000FF8C2BE680 00007ffd709c368f (MethodDesc 00007ffd70038a38 + 0x4f System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Raven.Server.RavenServer+<DispatchTcpConnection>d__79, Raven.Server]](<DispatchTcpConnection>d__79 ByRef)), calling (MethodDesc 00007ffd6ecd0180 + 0 Raven.Server.RavenServer+<DispatchTcpConnection>d__79.MoveNext())
000000FF8C2BE6E0 00007ffd709c35e2 (MethodDesc 00007ffd6dd70460 + 0x62 Raven.Server.RavenServer.DispatchTcpConnection(Raven.Client.ServerWide.Tcp.TcpConnectionHeaderMessage, Raven.Server.Documents.TcpHandlers.TcpConnectionOptions, MemoryBuffer, System.Security.Cryptography.X509Certificates.X509Certificate2)), calling (MethodDesc 00007ffd70038a38 + 0 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Raven.Server.RavenServer+<DispatchTcpConnection>d__79, Raven.Server]](<DispatchTcpConnection>d__79 ByRef))
000000FF8C2BE750 00007ffd7095af5c (MethodDesc 00007ffd6f33c908 + 0xafc Raven.Server.RavenServer+<>c__DisplayClass78_0+<<ListenToNewTcpConnection>b__0>d.MoveNext()), calling (MethodDesc 00007ffd6dd70460 + 0 Raven.Server.RavenServer.DispatchTcpConnection(Raven.Client.ServerWide.Tcp.TcpConnectionHeaderMessage, Raven.Server.Documents.TcpHandlers.TcpConnectionOptions, MemoryBuffer, System.Security.Cryptography.X509Certificates.X509Certificate2))
000000FF8C2BE780 00007ffd70937abe (MethodDesc 00007ffd6fcbb978 + 0x70e Sparrow.Json.AsyncBlittableJsonTextWriter+<DisposeAsync>d__8.MoveNext()), calling coreclr!JIT_CheckedWriteBarrier
000000FF8C2BE920 00007ffd7039a166 (MethodDesc 00007ffd6de8d4d0 + 0x76 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
000000FF8C2BE990 00007ffd7096d919 (MethodDesc 00007ffd6f59a800 + 0x59 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib],[Raven.Server.RavenServer+<>c__DisplayClass78_0+<<ListenToNewTcpConnection>b__0>d, Raven.Server]].MoveNext(System.Threading.Thread)), calling (MethodDesc 00007ffd6de8d4d0 + 0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
000000FF8C2BE9C0 00007ffdcd394fb0 coreclr!JIT_GetSharedGCThreadStaticBase + 0x70 [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1830], calling coreclr!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000FF8C2BEA00 00007ffd703f4f07 (MethodDesc 00007ffd6e5ad4b8 + 0xa7 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean))
000000FF8C2BEA70 00007ffd703f3db7 (MethodDesc 00007ffd6de549d0 + 0x77 System.Threading.Tasks.Task.RunContinuations(System.Object)), calling (MethodDesc 00007ffd6e5ad4b8 + 0 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean))
000000FF8C2BEA80 00007ffed3380443 ntdll!RtlGetSystemTimePrecise + 0x53, calling ntdll!RtlQueryPerformanceCounter
000000FF8C2BEAA0 00007ffd703e2bf9 (MethodDesc 00007ffd6e12d408 + 0x79 Sparrow.Json.PerCoreContainer`1[[System.__Canon, System.Private.CoreLib]].TryPush(System.__Canon)), calling 00007ffdcd442610 (stub for System.Threading.Interlocked.CompareExchange(System.Object ByRef, System.Object, System.Object))
000000FF8C2BEB10 00007ffd704da7d9 (MethodDesc 00007ffd6ecccc98 + 0xb9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.__Canon>, System.__Canon)), calling (MethodDesc 00007ffd6de549d0 + 0 System.Threading.Tasks.Task.RunContinuations(System.Object))
000000FF8C2BEB60 00007ffd7096f981 (MethodDesc 00007ffd6ecd08a0 + 0x1f21 Raven.Server.RavenServer+<NegotiateOperationVersion>d__80.MoveNext()), calling (MethodDesc 00007ffd6ecccc98 + 0 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.__Canon>, System.__Canon))
000000FF8C2BEB80 00007ffd703ffb96 (MethodDesc 00007ffd6fab16c8 + 0x456 Sparrow.Json.BlittableJsonDocumentBuilder.ReadInternal[[Sparrow.Json.BlittableJsonDocumentBuilder+WriteNone, Sparrow]]()), calling (MethodDesc 00007ffd6d97be18 + 0 System.Runtime.CompilerServices.CastHelpers.StelemRef(System.Array, IntPtr, System.Object))
000000FF8C2BEF80 00007ffd7090b396 (MethodDesc 00007ffd708e9cc0 + 0x26 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Raven.Server.RavenServer+<NegotiateOperationVersion>d__80, Raven.Server]].ExecutionContextCallback(System.Object)), calling (MethodDesc 00007ffd6ecd08a0 + 0 Raven.Server.RavenServer+<NegotiateOperationVersion>d__80.MoveNext())
000000FF8C2BEFB0 00007ffd7039a166 (MethodDesc 00007ffd6de8d4d0 + 0x76 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
000000FF8C2BF020 00007ffd7090b1d7 (MethodDesc 00007ffd708e9d00 + 0x137 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Raven.Server.RavenServer+<NegotiateOperationVersion>d__80, Raven.Server]].MoveNext(System.Threading.Thread)), calling (MethodDesc 00007ffd6de8d4d0 + 0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
000000FF8C2BF0B0 00007ffd70926f8a (MethodDesc 00007ffd708e9cf0 + 0x1a System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Raven.Server.RavenServer+<NegotiateOperationVersion>d__80, Raven.Server]].MoveNext()), calling 00007ffd70798828
000000FF8C2BF0E0 00007ffd703f4f07 (MethodDesc 00007ffd6e5ad4b8 + 0xa7 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean))
000000FF8C2BF150 00007ffd703f3db7 (MethodDesc 00007ffd6de549d0 + 0x77 System.Threading.Tasks.Task.RunContinuations(System.Object)), calling (MethodDesc 00007ffd6e5ad4b8 + 0 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean))
000000FF8C2BF180 00007ffd7043354b (MethodDesc 00007ffd6f45c250 + 0x4b Sparrow.Json.AbstractBlittableJsonDocumentBuilder.Dispose()), calling 00007ffd6d7c0010
000000FF8C2BF1F0 00007ffd704da7d9 (MethodDesc 00007ffd6ecccc98 + 0xb9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.__Canon>, System.__Canon)), calling (MethodDesc 00007ffd6de549d0 + 0 System.Threading.Tasks.Task.RunContinuations(System.Object))
000000FF8C2BF240 00007ffd709329c3 (MethodDesc 00007ffd6fd61188 + 0x24e3 Sparrow.Json.JsonOperationContext+<ParseToMemoryAsync>d__65.MoveNext()), calling (MethodDesc 00007ffd6ecccc98 + 0 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.__Canon>, System.__Canon))
000000FF8C2BF250 00007ffdcd394fb0 coreclr!JIT_GetSharedGCThreadStaticBase + 0x70 [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1830], calling coreclr!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000FF8C2BF6E0 00007ffd7039a166 (MethodDesc 00007ffd6de8d4d0 + 0x76 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
000000FF8C2BF750 00007ffd7099ee17 (MethodDesc 00007ffd7022d6b0 + 0x87 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Sparrow.Json.JsonOperationContext+<ParseToMemoryAsync>d__65, Sparrow]].MoveNext(System.Threading.Thread)), calling (MethodDesc 00007ffd6de8d4d0 + 0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
000000FF8C2BF760 00007ffdcd394fb0 coreclr!JIT_GetSharedGCThreadStaticBase + 0x70 [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1830], calling coreclr!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000FF8C2BF780 00007ffdcd37e9b3 coreclr!GCHandleManager::DestroyHandleOfUnknownType + 0x83 [D:\a\_work\1\s\src\coreclr\gc\gchandletable.cpp:129], calling coreclr!TableFreeSingleHandleToCache [D:\a\_work\1\s\src\coreclr\gc\handletablecache.cpp:768]
000000FF8C2BF7C0 00007ffd7091f137 (MethodDesc 00007ffd6f577d30 + 0x37 System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.InvokeContinuation(System.Action`1<System.Object>, System.Object, Boolean, Boolean))
000000FF8C2BF7F0 00007ffd70906670 (MethodDesc 00007ffd6f54aa50 + 0xa0 System.Net.Sockets.SocketAsyncEventArgs.<CompleteCore>g__CleanupIOCPResult|171_0()), calling (MethodDesc 00007ffd6e084d40 + 0 System.Threading.CancellationTokenSource+Registrations.Unregister(Int64, CallbackNode))
000000FF8C2BF830 00007ffd7091effc (MethodDesc 00007ffd6f577c10 + 0x8c System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.OnCompleted(System.Net.Sockets.SocketAsyncEventArgs)), calling (MethodDesc 00007ffd6f577d30 + 0 System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.InvokeContinuation(System.Action`1<System.Object>, System.Object, Boolean, Boolean))
000000FF8C2BF840 00007ffd7091fbb3 (MethodDesc 00007ffd6f54a5e8 + 0xd3 System.Net.Sockets.SocketAsyncEventArgs.FinishOperationSyncSuccess(Int32, System.Net.Sockets.SocketFlags)), calling (MethodDesc 00007ffd6f54aa50 + 0 System.Net.Sockets.SocketAsyncEventArgs.<CompleteCore>g__CleanupIOCPResult|171_0())
000000FF8C2BF850 00007ffd704d794b (MethodDesc 00007ffd6e53e8c0 + 0x1b System.Threading.ThreadPoolBoundHandle.GetOverlappedWrapper(System.Threading.NativeOverlapped*)), calling 00007ffdcd463a70 (stub for System.Threading.OverlappedData.GetOverlappedFromNative(System.Threading.NativeOverlapped*))
000000FF8C2BF8A0 00007ffd709063f0 (MethodDesc 00007ffd6f578c50 + 0x130 System.Net.Sockets.SocketAsyncEventArgs+<>c.<.cctor>b__176_0(UInt32, UInt32, System.Threading.NativeOverlapped*))
000000FF8C2BF8C0 00007ffd709061ab (MethodDesc 00007ffd6e5a9800 + 0x1b System.Threading.ThreadPoolBoundHandleOverlapped.CompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)), calling 00007ffdcd463a70 (stub for System.Threading.OverlappedData.GetOverlappedFromNative(System.Threading.NativeOverlapped*))
000000FF8C2BF900 00007ffd704daf5f (MethodDesc 00007ffd6e5515c0 + 0x1ff System.Threading.ThreadPoolTypedWorkItemQueue`2[[System.Threading.PortableThreadPool+IOCompletionPoller+Event, System.Private.CoreLib],[System.Threading.PortableThreadPool+IOCompletionPoller+Callback, System.Private.CoreLib]].System.Threading.IThreadPoolWorkItem.Execute())
000000FF8C2BF970 00007ffd703f450f (MethodDesc 00007ffd6f512320 + 0xaf System.Threading.PortableThreadPool+WorkerThread.MaybeAddWorkingWorker(System.Threading.PortableThreadPool)), calling (MethodDesc 00007ffd6f5124a0 + 0 System.Threading.LowLevelLifoSemaphore.Release(Int32))
000000FF8C2BF9F0 00007ffd703f47fa (MethodDesc 00007ffd6e53ec10 + 0x13a System.Threading.ThreadPoolWorkQueue.Dispatch())
000000FF8C2BFA08 00007ffd703f4c9e (MethodDesc 00007ffd6f512488 + 0xde System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean)), calling coreclr!ThreadNative_UninterruptibleSleep0 [D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp:479]
000000FF8C2BFA70 00007ffdcc09f2a7 (MethodDesc 00007ffd6f5122f0 + 0x177 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()), calling 00007ffd6e4ac768
000000FF8C2BFA90 00007ffed21720b9 combase!RegisterThreadCleanupCallback + 0x29 [onecore\com\combase\class\compobj.cxx:3139], calling KERNELBASE!FlsSetValue
000000FF8C2BFAE0 00007ffed2171d6b combase!_CoInitializeEx + 0x233 [onecore\com\combase\class\compobj.cxx:3725], calling combase!__security_check_cookie [VCCRT\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000FF8C2BFAF0 00007ffdcc09f2f5 (MethodDesc 00007ffd6f5122f0 + 0x1c5 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()), calling coreclr!JIT_PInvokeBegin
000000FF8C2BFB10 00007ffed217232b combase!CoUninitialize + 0x12b [onecore\com\combase\class\compobj.cxx:3908], calling combase!__security_check_cookie [VCCRT\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000FF8C2BFB80 00007ffdcd48fb73 coreclr!CallDescrWorkerInternal + 0x83
000000FF8C2BFBC0 00007ffdcd33eed0 coreclr!DispatchCallSimple + 0x60 [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:221], calling coreclr!CallDescrWorkerWithHandler [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:54]
000000FF8C2BFBD0 00007ffed2151c66 combase!RoInitialize + 0x26 [onecore\com\combase\winrtbase\winrtbase.cpp:351], calling combase!RoInitializeForApartmentType [onecore\com\combase\winrtbase\winrtbase.cpp:293]
000000FF8C2BFC50 00007ffdcd480583 coreclr!ThreadNative::KickOffThread_Worker + 0x63 [D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp:158], calling coreclr!DispatchCallSimple [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:171]
000000FF8C2BFCA0 00007ffdcd3e6911 coreclr!ETW::ThreadLog::FireThreadCreated + 0x39 [D:\a\_work\1\s\src\coreclr\vm\eventtrace.cpp:2599], calling coreclr!FireEtwThreadCreated [D:\a\_work\1\s\artifacts\obj\coreclr\windows.x64.Release\inc\clretwallmain.h:1677]
000000FF8C2BFCB0 00007ffdcd419e71 coreclr!ManagedThreadBase_DispatchMiddle + 0x85 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:7342], calling ntdll!LdrpDispatchUserCallTarget
000000FF8C2BFCC0 00007ffdcd3e66fc coreclr!ThreadStore::TransferStartedThread + 0x70 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:5468], calling coreclr!ThreadSuspend::UnlockThreadStore [D:\a\_work\1\s\src\coreclr\vm\threadsuspend.cpp:1914]
000000FF8C2BFCF0 00007ffdcd3e6af0 coreclr!Thread::HasStarted + 0xf4 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:1884], calling coreclr!CLRException::HandlerState::CleanupTry [D:\a\_work\1\s\src\coreclr\vm\clrex.cpp:791]
000000FF8C2BFD90 00007ffdcd419d87 coreclr!ManagedThreadBase_DispatchOuter + 0xab [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:7501], calling coreclr!ManagedThreadBase_DispatchMiddle [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:7302]
000000FF8C2BFE30 00007ffdcd419c79 coreclr!ThreadNative::KickOffThread + 0x79 [D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp:230], calling coreclr!ManagedThreadBase_DispatchOuter [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:7453]
000000FF8C2BFE90 00007ffed28226ad KERNEL32!BaseThreadInitThunk + 0x1d, calling KERNEL32!BaseThreadInitXfgThunk
000000FF8C2BFEC0 00007ffed33ca9f8 ntdll!RtlUserThreadStart + 0x28, calling ntdll!LdrpDispatchUserCallTarget

0:093> kp
 # Child-SP          RetAddr               Call Site
00 000000ff`8c2be200 00007ffd`cd45126a     coreclr!JIT_ClassInitDynamicClass_Helper(struct DomainLocalModule * pLocalModule = 0x00007ffd`6dbc0000, unsigned long dwDynamicClassDomainID = 0xffffffff)+0x7c [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp @ 1482] 
01 000000ff`8c2be360 00007ffd`70f4efc7     coreclr!JIT_ClassInitDynamicClass(struct DomainLocalModule * pLocalModule = <Value unavailable error>, unsigned long dwDynamicClassDomainID = <Value unavailable error>)+0x2a [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp @ 1503] 
02 000000ff`8c2be390 00007ffd`70f4e31e     Raven_Server!Raven.Server.RavenServer.CreateSubscriptionConnection+0x3e7
03 000000ff`8c2be430 00007ffd`70f4dbff     Raven_Server!Raven.Server.RavenServer.<DispatchDatabaseTcpConnection>d__99.MoveNext+0x64e
04 000000ff`8c2be520 00007ffd`709c38b0     System_Private_CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start<<DispatchDatabaseTcpConnection>d__99>+0x4f [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38] 
05 000000ff`8c2be580 00007ffd`709c368f     Raven_Server!Raven.Server.RavenServer.<DispatchTcpConnection>d__79.MoveNext+0x150
06 000000ff`8c2be690 00007ffd`709c35e2     0x00007ffd`709c368f
07 000000ff`8c2be6f0 00007ffd`7095af5c     Raven_Server!Raven.Server.RavenServer.DispatchTcpConnection+0x62
08 000000ff`8c2be760 00007ffd`7039a166     Raven_Server!Raven.Server.RavenServer.<>c__DisplayClass78_0.<<ListenToNewTcpConnection>b__0>d.MoveNext+0xafc
09 000000ff`8c2be930 00007ffd`7096d919     System_Private_CoreLib!System.Threading.ExecutionContext.RunInternal+0x76 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183] 
0a 000000ff`8c2be9a0 00007ffd`703f4f07     System_Private_CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<<<ListenToNewTcpConnection>b__0>d>.MoveNext+0x59 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 332] 
0b 000000ff`8c2bea10 00007ffd`703f3db7     System_Private_CoreLib!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction+0xa7 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs @ 795] 
0c 000000ff`8c2bea80 00007ffd`704da7d9     System_Private_CoreLib!System.Threading.Tasks.Task.RunContinuations+0x77 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 3411] 
0d 000000ff`8c2beb20 00007ffd`7096f981     System_Private_CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.SetExistingTaskResult+0xb9 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 446] 
0e 000000ff`8c2beb70 00007ffd`7090b396     Raven_Server!Raven.Server.RavenServer.<NegotiateOperationVersion>d__80.MoveNext+0x1f21
0f 000000ff`8c2bef90 00007ffd`7039a166     System_Private_CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<<NegotiateOperationVersion>d__80>.ExecutionContextCallback+0x26 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 287] 
10 000000ff`8c2befc0 00007ffd`7090b1d7     System_Private_CoreLib!System.Threading.ExecutionContext.RunInternal+0x76 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183] 
11 000000ff`8c2bf030 00007ffd`70926f8a     0x00007ffd`7090b1d7
12 000000ff`8c2bf0c0 00007ffd`703f4f07     System_Private_CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<<NegotiateOperationVersion>d__80>.MoveNext+0x1a [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 302] 
13 000000ff`8c2bf0f0 00007ffd`703f3db7     System_Private_CoreLib!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction+0xa7 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs @ 795] 
14 000000ff`8c2bf160 00007ffd`704da7d9     System_Private_CoreLib!System.Threading.Tasks.Task.RunContinuations+0x77 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 3411] 
15 000000ff`8c2bf200 00007ffd`709329c3     System_Private_CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.SetExistingTaskResult+0xb9 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 446] 
16 000000ff`8c2bf250 00007ffd`7039a166     Sparrow!Sparrow.Json.JsonOperationContext.<ParseToMemoryAsync>d__65.MoveNext+0x24e3
17 000000ff`8c2bf6f0 00007ffd`7099ee17     System_Private_CoreLib!System.Threading.ExecutionContext.RunInternal+0x76 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183] 
18 000000ff`8c2bf760 00007ffd`7091f137     0x00007ffd`7099ee17
19 000000ff`8c2bf7d0 00007ffd`7091effc     System_Net_Sockets!System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.InvokeContinuation+0x37 [/_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs @ 1321] 
1a 000000ff`8c2bf840 00007ffd`709063f0     System_Net_Sockets!System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.OnCompleted+0x8c [/_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs @ 1004] 
1b 000000ff`8c2bf8b0 00007ffd`704daf5f     System_Net_Sockets!System.Net.Sockets.SocketAsyncEventArgs.<>c.<.cctor>b__176_0+0x130 [/_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Windows.cs @ 1250] 
1c 000000ff`8c2bf910 00007ffd`703f47fa     System_Private_CoreLib!System.Threading.ThreadPoolTypedWorkItemQueue<Event,Callback>.System.Threading.IThreadPoolWorkItem.Execute+0x1ff [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs @ 1143] 
1d 000000ff`8c2bfa00 00007ffd`cc09f2a7     System_Private_CoreLib!System.Threading.ThreadPoolWorkQueue.Dispatch+0x13a [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs @ 924] 
1e 000000ff`8c2bfa80 00007ffd`cd48fb73     System_Private_CoreLib!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart(void)+0x177 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 77] 
1f 000000ff`8c2bfb90 00007ffd`cd33eed0     coreclr!CallDescrWorkerInternal+0x83
20 000000ff`8c2bfbd0 00007ffd`cd480583     coreclr!DispatchCallSimple(unsigned int64 * pSrc = 0x000000ff`8c2bfc80, unsigned long numStackSlotsToCopy = 4, unsigned int64 pTargetAddress = 0x00007ffd`6da26150, unsigned long dwDispatchCallSimpleFlags = 0)+0x60 [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp @ 221] 
21 000000ff`8c2bfc60 00007ffd`cd419e71     coreclr!ThreadNative::KickOffThread_Worker(void * ptr = <Value unavailable error>)+0x63 [D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp @ 158] 
22 (Inline Function) --------`--------     coreclr!ManagedThreadBase_DispatchInner(void)+0xd [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7298] 
23 000000ff`8c2bfcc0 00007ffd`cd419d87     coreclr!ManagedThreadBase_DispatchMiddle(struct ManagedThreadCallState * pCallState = 0x000000ff`8c2bfd38)+0x85 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7342] 
24 000000ff`8c2bfda0 00007ffd`cd419c79     coreclr!ManagedThreadBase_DispatchOuter(struct ManagedThreadCallState * pCallState = <Value unavailable error>)+0xab [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7501] 
25 (Inline Function) --------`--------     coreclr!ManagedThreadBase_FullTransition(void)+0x2d [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7546] 
26 (Inline Function) --------`--------     coreclr!ManagedThreadBase::KickOff(void)+0x2d [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7581] 
27 000000ff`8c2bfe40 00007ffe`d28226ad     coreclr!ThreadNative::KickOffThread(void * pass = 0x0000029c`8df4e370)+0x79 [D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp @ 230] 
28 000000ff`8c2bfea0 00007ffe`d33ca9f8     KERNEL32!BaseThreadInitThunk+0x1d
29 000000ff`8c2bfed0 00000000`00000000     ntdll!RtlUserThreadStart+0x28

Heap

There is no heap corruption:

0:093> !verifyheap
3,822,041 objects verified, 0 errors.
No heap corruption detected.

RavenDB code

Reference to our code that is shown in the stacktrace:
https://github.com/ravendb/ravendb/blob/9f86bd3d3b08e07f67a5a238bd8dd77933774765/src/Raven.Server/RavenServer.cs#L2491

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

We have a scenario in which we are getting an access violation exception during a particular state of our RavenDB cluster when Dynamic PGO is enabled.

The setup is 3 servers, 2 of them are down and a client app is constantly trying to connect via TCP to the running server.
The connections are established but then some of the connections are dropped because they were supposed to by handled by a different node (which is down).

We have a way to reproduce it. The key seems to be that it is happening only with Dynamic PGO enabled. Disabling it makes that no crash is experienced.

We have a full dump of the process when the crash happened, available here:
https://drive.google.com/file/d/1EC0Gwz_ljWuCdJX2WaKDxfdGMpoINnba/view?usp=sharing

We're running using:

Architecture: x64
RuntimeFrameworkVersion: 7.0.7
TieredPGO: True

Exception

(72b4.5ee4): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
coreclr!JIT_ClassInitDynamicClass_Helper+0x7c:
00007ffd`cd4512e8 488b0cc8        mov     rcx,qword ptr [rax+rcx*8] ds:00008005`7003e060=????????????????

Stacktrace

0:093> !clrstack
OS Thread Id: 0x5ee4 (93)
        Child SP               IP Call Site
000000FF8C2BE258 00007ffdcd4512e8 [HelperMethodFrame: 000000ff8c2be258] 
000000FF8C2BE390 00007ffd70f4efc7 Raven.Server.RavenServer.CreateSubscriptionConnection(Raven.Server.ServerWide.ServerStore, DatabaseSearchResult, Raven.Server.Documents.TcpHandlers.TcpConnectionOptions, MemoryBuffer)
000000FF8C2BE430 00007ffd70f4e31e Raven.Server.RavenServer+d__99.MoveNext() [D:\workspace\ravendb-6.0\src\Raven.Server\RavenServer.cs @ 2463]
000000FF8C2BE520 00007ffd70f4dbff System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Raven.Server.RavenServer+d__99, Raven.Server]](d__99 ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
000000FF8C2BE580 00007ffd709c38b0 Raven.Server.RavenServer+d__79.MoveNext() [D:\workspace\ravendb-6.0\src\Raven.Server\RavenServer.cs @ 2074]
000000FF8C2BE690 00007ffd709c368f System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Raven.Server.RavenServer+d__79, Raven.Server]](d__79 ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
000000FF8C2BE6F0 00007ffd709c35e2 Raven.Server.RavenServer.DispatchTcpConnection(Raven.Client.ServerWide.Tcp.TcpConnectionHeaderMessage, Raven.Server.Documents.TcpHandlers.TcpConnectionOptions, MemoryBuffer, System.Security.Cryptography.X509Certificates.X509Certificate2)
000000FF8C2BE760 00007ffd7095af5c Raven.Server.RavenServer+c__DisplayClass78_0+<b__0>d.MoveNext() [D:\workspace\ravendb-6.0\src\Raven.Server\RavenServer.cs @ 2006]
000000FF8C2BE930 00007ffd7039a166 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183]
000000FF8C2BE9A0 00007ffd7096d919 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib],[Raven.Server.RavenServer+c__DisplayClass78_0+<b__0>d, Raven.Server]].MoveNext(System.Threading.Thread) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 324]
000000FF8C2BEA10 00007ffd703f4f07 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs @ 795]
000000FF8C2BEA80 00007ffd703f3db7 System.Threading.Tasks.Task.RunContinuations(System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 3411]
000000FF8C2BEB20 00007ffd704da7d9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.__Canon>, System.__Canon) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 446]
000000FF8C2BEB70 00007ffd7096f981 Raven.Server.RavenServer+d__80.MoveNext() [D:\workspace\ravendb-6.0\src\Raven.Server\RavenServer.cs @ 2208]
000000FF8C2BEF90 00007ffd7090b396 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Raven.Server.RavenServer+d__80, Raven.Server]].ExecutionContextCallback(System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 286]
000000FF8C2BEFC0 00007ffd7039a166 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183]
000000FF8C2BF030 00007ffd7090b1d7 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Raven.Server.RavenServer+d__80, Raven.Server]].MoveNext(System.Threading.Thread) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 324]
000000FF8C2BF0C0 00007ffd70926f8a System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Raven.Server.RavenServer+d__80, Raven.Server]].MoveNext() [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 302]
000000FF8C2BF0F0 00007ffd703f4f07 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs @ 795]
000000FF8C2BF160 00007ffd703f3db7 System.Threading.Tasks.Task.RunContinuations(System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 3411]
000000FF8C2BF200 00007ffd704da7d9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.__Canon>, System.__Canon) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 446]
000000FF8C2BF250 00007ffd709329c3 Sparrow.Json.JsonOperationContext+d__65.MoveNext() [D:\workspace\ravendb-6.0\src\Sparrow\Json\JsonOperationContext.cs @ 835]
000000FF8C2BF6F0 00007ffd7039a166 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183]
000000FF8C2BF760 00007ffd7099ee17 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Sparrow.Json.JsonOperationContext+d__65, Sparrow]].MoveNext(System.Threading.Thread) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 324]
000000FF8C2BF7D0 00007ffd7091f137 System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.InvokeContinuation(System.Action`1<System.Object>, System.Object, Boolean, Boolean) [/_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs @ 1321]
000000FF8C2BF840 00007ffd7091effc System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.OnCompleted(System.Net.Sockets.SocketAsyncEventArgs) [/_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs @ 1004]
000000FF8C2BF8B0 00007ffd709063f0 System.Net.Sockets.SocketAsyncEventArgs+c.<.cctor>b__176_0(UInt32, UInt32, System.Threading.NativeOverlapped*) [/_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Windows.cs @ 1250]
000000FF8C2BF910 00007ffd704daf5f System.Threading.ThreadPoolTypedWorkItemQueue`2[[System.Threading.PortableThreadPool+IOCompletionPoller+Event, System.Private.CoreLib],[System.Threading.PortableThreadPool+IOCompletionPoller+Callback, System.Private.CoreLib]].System.Threading.IThreadPoolWorkItem.Execute() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs @ 1134]
000000FF8C2BFA00 00007ffd703f47fa System.Threading.ThreadPoolWorkQueue.Dispatch() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs @ 924]
000000FF8C2BFA80 00007ffdcc09f2a7 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 77]
000000FF8C2BFDE0 00007ffdcd48fb73 [DebuggerU2MCatchHandlerFrame: 000000ff8c2bfde0] 

0:093> !dumpstack
OS Thread Id: 0x5ee4 (93)
Current frame: coreclr!JIT_ClassInitDynamicClass_Helper + 0x7c [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1482]
Child-SP         RetAddr          Caller, Callee
000000FF8C2BE260 00007ffd70f505f8 (MethodDesc 00007ffd708ecfe0 + 0x78 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Raven.Server.Documents.Subscriptions.SubscriptionBinder`3+<Run>d__6[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]], Raven.Server]](<Run>d__6<System.__Canon,System.__Canon,System.__Canon> ByRef))
000000FF8C2BE2D0 00007ffd70f504dc (MethodDesc 00007ffd708eae70 + 0xac Raven.Server.Documents.Subscriptions.SubscriptionBinder`3[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].Run(Raven.Server.Documents.TcpHandlers.TcpConnectionOptions, System.IDisposable)), calling (MethodDesc 00007ffd708ecd20 + 0 System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Raven.Server.Documents.Subscriptions.SubscriptionBinder`3+<Run>d__6[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]], Raven.Server]](<Run>d__6<System.__Canon,System.__Canon,System.__Canon> ByRef))
000000FF8C2BE318 00007ffdcd4512c9 coreclr!JIT_ClassInitDynamicClass_Helper + 0x5d [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1477], calling coreclr!LazyMachStateCaptureState
000000FF8C2BE350 00007ffdcd45126a coreclr!JIT_ClassInitDynamicClass + 0x2a [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1503], calling coreclr!JIT_ClassInitDynamicClass_Helper [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1473]
000000FF8C2BE380 00007ffd70f4efc7 (MethodDesc 00007ffd6dd70550 + 0x3e7 Raven.Server.RavenServer.CreateSubscriptionConnection(Raven.Server.ServerWide.ServerStore, DatabaseSearchResult, Raven.Server.Documents.TcpHandlers.TcpConnectionOptions, MemoryBuffer)), calling coreclr!JIT_ClassInitDynamicClass [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1491]
000000FF8C2BE420 00007ffd70f4e31e (MethodDesc 00007ffd6ecce520 + 0x64e Raven.Server.RavenServer+<DispatchDatabaseTcpConnection>d__99.MoveNext()), calling (MethodDesc 00007ffd6dd70550 + 0 Raven.Server.RavenServer.CreateSubscriptionConnection(Raven.Server.ServerWide.ServerStore, DatabaseSearchResult, Raven.Server.Documents.TcpHandlers.TcpConnectionOptions, MemoryBuffer))
000000FF8C2BE510 00007ffd70f4dbff (MethodDesc 00007ffd708e9c30 + 0x4f System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Raven.Server.RavenServer+<DispatchDatabaseTcpConnection>d__99, Raven.Server]](<DispatchDatabaseTcpConnection>d__99 ByRef)), calling (MethodDesc 00007ffd6ecce520 + 0 Raven.Server.RavenServer+<DispatchDatabaseTcpConnection>d__99.MoveNext())
000000FF8C2BE520 00007ffd704e845f (MethodDesc 00007ffd6f577bf8 + 0x4f System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.Release()), calling 00007ffdcd442610 (stub for System.Threading.Interlocked.CompareExchange(System.Object ByRef, System.Object, System.Object))
000000FF8C2BE570 00007ffd709c38b0 (MethodDesc 00007ffd6ecd0180 + 0x150 Raven.Server.RavenServer+<DispatchTcpConnection>d__79.MoveNext()), calling (MethodDesc 00007ffd708e9c30 + 0 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Raven.Server.RavenServer+<DispatchDatabaseTcpConnection>d__99, Raven.Server]](<DispatchDatabaseTcpConnection>d__99 ByRef))
000000FF8C2BE680 00007ffd709c368f (MethodDesc 00007ffd70038a38 + 0x4f System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Raven.Server.RavenServer+<DispatchTcpConnection>d__79, Raven.Server]](<DispatchTcpConnection>d__79 ByRef)), calling (MethodDesc 00007ffd6ecd0180 + 0 Raven.Server.RavenServer+<DispatchTcpConnection>d__79.MoveNext())
000000FF8C2BE6E0 00007ffd709c35e2 (MethodDesc 00007ffd6dd70460 + 0x62 Raven.Server.RavenServer.DispatchTcpConnection(Raven.Client.ServerWide.Tcp.TcpConnectionHeaderMessage, Raven.Server.Documents.TcpHandlers.TcpConnectionOptions, MemoryBuffer, System.Security.Cryptography.X509Certificates.X509Certificate2)), calling (MethodDesc 00007ffd70038a38 + 0 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Raven.Server.RavenServer+<DispatchTcpConnection>d__79, Raven.Server]](<DispatchTcpConnection>d__79 ByRef))
000000FF8C2BE750 00007ffd7095af5c (MethodDesc 00007ffd6f33c908 + 0xafc Raven.Server.RavenServer+<>c__DisplayClass78_0+<<ListenToNewTcpConnection>b__0>d.MoveNext()), calling (MethodDesc 00007ffd6dd70460 + 0 Raven.Server.RavenServer.DispatchTcpConnection(Raven.Client.ServerWide.Tcp.TcpConnectionHeaderMessage, Raven.Server.Documents.TcpHandlers.TcpConnectionOptions, MemoryBuffer, System.Security.Cryptography.X509Certificates.X509Certificate2))
000000FF8C2BE780 00007ffd70937abe (MethodDesc 00007ffd6fcbb978 + 0x70e Sparrow.Json.AsyncBlittableJsonTextWriter+<DisposeAsync>d__8.MoveNext()), calling coreclr!JIT_CheckedWriteBarrier
000000FF8C2BE920 00007ffd7039a166 (MethodDesc 00007ffd6de8d4d0 + 0x76 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
000000FF8C2BE990 00007ffd7096d919 (MethodDesc 00007ffd6f59a800 + 0x59 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib],[Raven.Server.RavenServer+<>c__DisplayClass78_0+<<ListenToNewTcpConnection>b__0>d, Raven.Server]].MoveNext(System.Threading.Thread)), calling (MethodDesc 00007ffd6de8d4d0 + 0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
000000FF8C2BE9C0 00007ffdcd394fb0 coreclr!JIT_GetSharedGCThreadStaticBase + 0x70 [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1830], calling coreclr!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000FF8C2BEA00 00007ffd703f4f07 (MethodDesc 00007ffd6e5ad4b8 + 0xa7 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean))
000000FF8C2BEA70 00007ffd703f3db7 (MethodDesc 00007ffd6de549d0 + 0x77 System.Threading.Tasks.Task.RunContinuations(System.Object)), calling (MethodDesc 00007ffd6e5ad4b8 + 0 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean))
000000FF8C2BEA80 00007ffed3380443 ntdll!RtlGetSystemTimePrecise + 0x53, calling ntdll!RtlQueryPerformanceCounter
000000FF8C2BEAA0 00007ffd703e2bf9 (MethodDesc 00007ffd6e12d408 + 0x79 Sparrow.Json.PerCoreContainer`1[[System.__Canon, System.Private.CoreLib]].TryPush(System.__Canon)), calling 00007ffdcd442610 (stub for System.Threading.Interlocked.CompareExchange(System.Object ByRef, System.Object, System.Object))
000000FF8C2BEB10 00007ffd704da7d9 (MethodDesc 00007ffd6ecccc98 + 0xb9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.__Canon>, System.__Canon)), calling (MethodDesc 00007ffd6de549d0 + 0 System.Threading.Tasks.Task.RunContinuations(System.Object))
000000FF8C2BEB60 00007ffd7096f981 (MethodDesc 00007ffd6ecd08a0 + 0x1f21 Raven.Server.RavenServer+<NegotiateOperationVersion>d__80.MoveNext()), calling (MethodDesc 00007ffd6ecccc98 + 0 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.__Canon>, System.__Canon))
000000FF8C2BEB80 00007ffd703ffb96 (MethodDesc 00007ffd6fab16c8 + 0x456 Sparrow.Json.BlittableJsonDocumentBuilder.ReadInternal[[Sparrow.Json.BlittableJsonDocumentBuilder+WriteNone, Sparrow]]()), calling (MethodDesc 00007ffd6d97be18 + 0 System.Runtime.CompilerServices.CastHelpers.StelemRef(System.Array, IntPtr, System.Object))
000000FF8C2BEF80 00007ffd7090b396 (MethodDesc 00007ffd708e9cc0 + 0x26 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Raven.Server.RavenServer+<NegotiateOperationVersion>d__80, Raven.Server]].ExecutionContextCallback(System.Object)), calling (MethodDesc 00007ffd6ecd08a0 + 0 Raven.Server.RavenServer+<NegotiateOperationVersion>d__80.MoveNext())
000000FF8C2BEFB0 00007ffd7039a166 (MethodDesc 00007ffd6de8d4d0 + 0x76 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
000000FF8C2BF020 00007ffd7090b1d7 (MethodDesc 00007ffd708e9d00 + 0x137 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Raven.Server.RavenServer+<NegotiateOperationVersion>d__80, Raven.Server]].MoveNext(System.Threading.Thread)), calling (MethodDesc 00007ffd6de8d4d0 + 0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
000000FF8C2BF0B0 00007ffd70926f8a (MethodDesc 00007ffd708e9cf0 + 0x1a System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Raven.Server.RavenServer+<NegotiateOperationVersion>d__80, Raven.Server]].MoveNext()), calling 00007ffd70798828
000000FF8C2BF0E0 00007ffd703f4f07 (MethodDesc 00007ffd6e5ad4b8 + 0xa7 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean))
000000FF8C2BF150 00007ffd703f3db7 (MethodDesc 00007ffd6de549d0 + 0x77 System.Threading.Tasks.Task.RunContinuations(System.Object)), calling (MethodDesc 00007ffd6e5ad4b8 + 0 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean))
000000FF8C2BF180 00007ffd7043354b (MethodDesc 00007ffd6f45c250 + 0x4b Sparrow.Json.AbstractBlittableJsonDocumentBuilder.Dispose()), calling 00007ffd6d7c0010
000000FF8C2BF1F0 00007ffd704da7d9 (MethodDesc 00007ffd6ecccc98 + 0xb9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.__Canon>, System.__Canon)), calling (MethodDesc 00007ffd6de549d0 + 0 System.Threading.Tasks.Task.RunContinuations(System.Object))
000000FF8C2BF240 00007ffd709329c3 (MethodDesc 00007ffd6fd61188 + 0x24e3 Sparrow.Json.JsonOperationContext+<ParseToMemoryAsync>d__65.MoveNext()), calling (MethodDesc 00007ffd6ecccc98 + 0 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.__Canon>, System.__Canon))
000000FF8C2BF250 00007ffdcd394fb0 coreclr!JIT_GetSharedGCThreadStaticBase + 0x70 [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1830], calling coreclr!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000FF8C2BF6E0 00007ffd7039a166 (MethodDesc 00007ffd6de8d4d0 + 0x76 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
000000FF8C2BF750 00007ffd7099ee17 (MethodDesc 00007ffd7022d6b0 + 0x87 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Sparrow.Json.JsonOperationContext+<ParseToMemoryAsync>d__65, Sparrow]].MoveNext(System.Threading.Thread)), calling (MethodDesc 00007ffd6de8d4d0 + 0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
000000FF8C2BF760 00007ffdcd394fb0 coreclr!JIT_GetSharedGCThreadStaticBase + 0x70 [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp:1830], calling coreclr!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000FF8C2BF780 00007ffdcd37e9b3 coreclr!GCHandleManager::DestroyHandleOfUnknownType + 0x83 [D:\a\_work\1\s\src\coreclr\gc\gchandletable.cpp:129], calling coreclr!TableFreeSingleHandleToCache [D:\a\_work\1\s\src\coreclr\gc\handletablecache.cpp:768]
000000FF8C2BF7C0 00007ffd7091f137 (MethodDesc 00007ffd6f577d30 + 0x37 System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.InvokeContinuation(System.Action`1<System.Object>, System.Object, Boolean, Boolean))
000000FF8C2BF7F0 00007ffd70906670 (MethodDesc 00007ffd6f54aa50 + 0xa0 System.Net.Sockets.SocketAsyncEventArgs.<CompleteCore>g__CleanupIOCPResult|171_0()), calling (MethodDesc 00007ffd6e084d40 + 0 System.Threading.CancellationTokenSource+Registrations.Unregister(Int64, CallbackNode))
000000FF8C2BF830 00007ffd7091effc (MethodDesc 00007ffd6f577c10 + 0x8c System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.OnCompleted(System.Net.Sockets.SocketAsyncEventArgs)), calling (MethodDesc 00007ffd6f577d30 + 0 System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.InvokeContinuation(System.Action`1<System.Object>, System.Object, Boolean, Boolean))
000000FF8C2BF840 00007ffd7091fbb3 (MethodDesc 00007ffd6f54a5e8 + 0xd3 System.Net.Sockets.SocketAsyncEventArgs.FinishOperationSyncSuccess(Int32, System.Net.Sockets.SocketFlags)), calling (MethodDesc 00007ffd6f54aa50 + 0 System.Net.Sockets.SocketAsyncEventArgs.<CompleteCore>g__CleanupIOCPResult|171_0())
000000FF8C2BF850 00007ffd704d794b (MethodDesc 00007ffd6e53e8c0 + 0x1b System.Threading.ThreadPoolBoundHandle.GetOverlappedWrapper(System.Threading.NativeOverlapped*)), calling 00007ffdcd463a70 (stub for System.Threading.OverlappedData.GetOverlappedFromNative(System.Threading.NativeOverlapped*))
000000FF8C2BF8A0 00007ffd709063f0 (MethodDesc 00007ffd6f578c50 + 0x130 System.Net.Sockets.SocketAsyncEventArgs+<>c.<.cctor>b__176_0(UInt32, UInt32, System.Threading.NativeOverlapped*))
000000FF8C2BF8C0 00007ffd709061ab (MethodDesc 00007ffd6e5a9800 + 0x1b System.Threading.ThreadPoolBoundHandleOverlapped.CompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)), calling 00007ffdcd463a70 (stub for System.Threading.OverlappedData.GetOverlappedFromNative(System.Threading.NativeOverlapped*))
000000FF8C2BF900 00007ffd704daf5f (MethodDesc 00007ffd6e5515c0 + 0x1ff System.Threading.ThreadPoolTypedWorkItemQueue`2[[System.Threading.PortableThreadPool+IOCompletionPoller+Event, System.Private.CoreLib],[System.Threading.PortableThreadPool+IOCompletionPoller+Callback, System.Private.CoreLib]].System.Threading.IThreadPoolWorkItem.Execute())
000000FF8C2BF970 00007ffd703f450f (MethodDesc 00007ffd6f512320 + 0xaf System.Threading.PortableThreadPool+WorkerThread.MaybeAddWorkingWorker(System.Threading.PortableThreadPool)), calling (MethodDesc 00007ffd6f5124a0 + 0 System.Threading.LowLevelLifoSemaphore.Release(Int32))
000000FF8C2BF9F0 00007ffd703f47fa (MethodDesc 00007ffd6e53ec10 + 0x13a System.Threading.ThreadPoolWorkQueue.Dispatch())
000000FF8C2BFA08 00007ffd703f4c9e (MethodDesc 00007ffd6f512488 + 0xde System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean)), calling coreclr!ThreadNative_UninterruptibleSleep0 [D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp:479]
000000FF8C2BFA70 00007ffdcc09f2a7 (MethodDesc 00007ffd6f5122f0 + 0x177 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()), calling 00007ffd6e4ac768
000000FF8C2BFA90 00007ffed21720b9 combase!RegisterThreadCleanupCallback + 0x29 [onecore\com\combase\class\compobj.cxx:3139], calling KERNELBASE!FlsSetValue
000000FF8C2BFAE0 00007ffed2171d6b combase!_CoInitializeEx + 0x233 [onecore\com\combase\class\compobj.cxx:3725], calling combase!__security_check_cookie [VCCRT\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000FF8C2BFAF0 00007ffdcc09f2f5 (MethodDesc 00007ffd6f5122f0 + 0x1c5 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()), calling coreclr!JIT_PInvokeBegin
000000FF8C2BFB10 00007ffed217232b combase!CoUninitialize + 0x12b [onecore\com\combase\class\compobj.cxx:3908], calling combase!__security_check_cookie [VCCRT\vcstartup\src\gs\amd64\amdsecgs.asm:45]
000000FF8C2BFB80 00007ffdcd48fb73 coreclr!CallDescrWorkerInternal + 0x83
000000FF8C2BFBC0 00007ffdcd33eed0 coreclr!DispatchCallSimple + 0x60 [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:221], calling coreclr!CallDescrWorkerWithHandler [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:54]
000000FF8C2BFBD0 00007ffed2151c66 combase!RoInitialize + 0x26 [onecore\com\combase\winrtbase\winrtbase.cpp:351], calling combase!RoInitializeForApartmentType [onecore\com\combase\winrtbase\winrtbase.cpp:293]
000000FF8C2BFC50 00007ffdcd480583 coreclr!ThreadNative::KickOffThread_Worker + 0x63 [D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp:158], calling coreclr!DispatchCallSimple [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:171]
000000FF8C2BFCA0 00007ffdcd3e6911 coreclr!ETW::ThreadLog::FireThreadCreated + 0x39 [D:\a\_work\1\s\src\coreclr\vm\eventtrace.cpp:2599], calling coreclr!FireEtwThreadCreated [D:\a\_work\1\s\artifacts\obj\coreclr\windows.x64.Release\inc\clretwallmain.h:1677]
000000FF8C2BFCB0 00007ffdcd419e71 coreclr!ManagedThreadBase_DispatchMiddle + 0x85 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:7342], calling ntdll!LdrpDispatchUserCallTarget
000000FF8C2BFCC0 00007ffdcd3e66fc coreclr!ThreadStore::TransferStartedThread + 0x70 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:5468], calling coreclr!ThreadSuspend::UnlockThreadStore [D:\a\_work\1\s\src\coreclr\vm\threadsuspend.cpp:1914]
000000FF8C2BFCF0 00007ffdcd3e6af0 coreclr!Thread::HasStarted + 0xf4 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:1884], calling coreclr!CLRException::HandlerState::CleanupTry [D:\a\_work\1\s\src\coreclr\vm\clrex.cpp:791]
000000FF8C2BFD90 00007ffdcd419d87 coreclr!ManagedThreadBase_DispatchOuter + 0xab [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:7501], calling coreclr!ManagedThreadBase_DispatchMiddle [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:7302]
000000FF8C2BFE30 00007ffdcd419c79 coreclr!ThreadNative::KickOffThread + 0x79 [D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp:230], calling coreclr!ManagedThreadBase_DispatchOuter [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:7453]
000000FF8C2BFE90 00007ffed28226ad KERNEL32!BaseThreadInitThunk + 0x1d, calling KERNEL32!BaseThreadInitXfgThunk
000000FF8C2BFEC0 00007ffed33ca9f8 ntdll!RtlUserThreadStart + 0x28, calling ntdll!LdrpDispatchUserCallTarget

0:093> kp
 # Child-SP          RetAddr               Call Site
00 000000ff`8c2be200 00007ffd`cd45126a     coreclr!JIT_ClassInitDynamicClass_Helper(struct DomainLocalModule * pLocalModule = 0x00007ffd`6dbc0000, unsigned long dwDynamicClassDomainID = 0xffffffff)+0x7c [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp @ 1482] 
01 000000ff`8c2be360 00007ffd`70f4efc7     coreclr!JIT_ClassInitDynamicClass(struct DomainLocalModule * pLocalModule = <Value unavailable error>, unsigned long dwDynamicClassDomainID = <Value unavailable error>)+0x2a [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp @ 1503] 
02 000000ff`8c2be390 00007ffd`70f4e31e     Raven_Server!Raven.Server.RavenServer.CreateSubscriptionConnection+0x3e7
03 000000ff`8c2be430 00007ffd`70f4dbff     Raven_Server!Raven.Server.RavenServer.<DispatchDatabaseTcpConnection>d__99.MoveNext+0x64e
04 000000ff`8c2be520 00007ffd`709c38b0     System_Private_CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start<<DispatchDatabaseTcpConnection>d__99>+0x4f [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38] 
05 000000ff`8c2be580 00007ffd`709c368f     Raven_Server!Raven.Server.RavenServer.<DispatchTcpConnection>d__79.MoveNext+0x150
06 000000ff`8c2be690 00007ffd`709c35e2     0x00007ffd`709c368f
07 000000ff`8c2be6f0 00007ffd`7095af5c     Raven_Server!Raven.Server.RavenServer.DispatchTcpConnection+0x62
08 000000ff`8c2be760 00007ffd`7039a166     Raven_Server!Raven.Server.RavenServer.<>c__DisplayClass78_0.<<ListenToNewTcpConnection>b__0>d.MoveNext+0xafc
09 000000ff`8c2be930 00007ffd`7096d919     System_Private_CoreLib!System.Threading.ExecutionContext.RunInternal+0x76 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183] 
0a 000000ff`8c2be9a0 00007ffd`703f4f07     System_Private_CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<<<ListenToNewTcpConnection>b__0>d>.MoveNext+0x59 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 332] 
0b 000000ff`8c2bea10 00007ffd`703f3db7     System_Private_CoreLib!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction+0xa7 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs @ 795] 
0c 000000ff`8c2bea80 00007ffd`704da7d9     System_Private_CoreLib!System.Threading.Tasks.Task.RunContinuations+0x77 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 3411] 
0d 000000ff`8c2beb20 00007ffd`7096f981     System_Private_CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.SetExistingTaskResult+0xb9 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 446] 
0e 000000ff`8c2beb70 00007ffd`7090b396     Raven_Server!Raven.Server.RavenServer.<NegotiateOperationVersion>d__80.MoveNext+0x1f21
0f 000000ff`8c2bef90 00007ffd`7039a166     System_Private_CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<<NegotiateOperationVersion>d__80>.ExecutionContextCallback+0x26 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 287] 
10 000000ff`8c2befc0 00007ffd`7090b1d7     System_Private_CoreLib!System.Threading.ExecutionContext.RunInternal+0x76 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183] 
11 000000ff`8c2bf030 00007ffd`70926f8a     0x00007ffd`7090b1d7
12 000000ff`8c2bf0c0 00007ffd`703f4f07     System_Private_CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<<NegotiateOperationVersion>d__80>.MoveNext+0x1a [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 302] 
13 000000ff`8c2bf0f0 00007ffd`703f3db7     System_Private_CoreLib!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction+0xa7 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs @ 795] 
14 000000ff`8c2bf160 00007ffd`704da7d9     System_Private_CoreLib!System.Threading.Tasks.Task.RunContinuations+0x77 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 3411] 
15 000000ff`8c2bf200 00007ffd`709329c3     System_Private_CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.SetExistingTaskResult+0xb9 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 446] 
16 000000ff`8c2bf250 00007ffd`7039a166     Sparrow!Sparrow.Json.JsonOperationContext.<ParseToMemoryAsync>d__65.MoveNext+0x24e3
17 000000ff`8c2bf6f0 00007ffd`7099ee17     System_Private_CoreLib!System.Threading.ExecutionContext.RunInternal+0x76 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183] 
18 000000ff`8c2bf760 00007ffd`7091f137     0x00007ffd`7099ee17
19 000000ff`8c2bf7d0 00007ffd`7091effc     System_Net_Sockets!System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.InvokeContinuation+0x37 [/_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs @ 1321] 
1a 000000ff`8c2bf840 00007ffd`709063f0     System_Net_Sockets!System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.OnCompleted+0x8c [/_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs @ 1004] 
1b 000000ff`8c2bf8b0 00007ffd`704daf5f     System_Net_Sockets!System.Net.Sockets.SocketAsyncEventArgs.<>c.<.cctor>b__176_0+0x130 [/_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Windows.cs @ 1250] 
1c 000000ff`8c2bf910 00007ffd`703f47fa     System_Private_CoreLib!System.Threading.ThreadPoolTypedWorkItemQueue<Event,Callback>.System.Threading.IThreadPoolWorkItem.Execute+0x1ff [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs @ 1143] 
1d 000000ff`8c2bfa00 00007ffd`cc09f2a7     System_Private_CoreLib!System.Threading.ThreadPoolWorkQueue.Dispatch+0x13a [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs @ 924] 
1e 000000ff`8c2bfa80 00007ffd`cd48fb73     System_Private_CoreLib!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart(void)+0x177 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs @ 77] 
1f 000000ff`8c2bfb90 00007ffd`cd33eed0     coreclr!CallDescrWorkerInternal+0x83
20 000000ff`8c2bfbd0 00007ffd`cd480583     coreclr!DispatchCallSimple(unsigned int64 * pSrc = 0x000000ff`8c2bfc80, unsigned long numStackSlotsToCopy = 4, unsigned int64 pTargetAddress = 0x00007ffd`6da26150, unsigned long dwDispatchCallSimpleFlags = 0)+0x60 [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp @ 221] 
21 000000ff`8c2bfc60 00007ffd`cd419e71     coreclr!ThreadNative::KickOffThread_Worker(void * ptr = <Value unavailable error>)+0x63 [D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp @ 158] 
22 (Inline Function) --------`--------     coreclr!ManagedThreadBase_DispatchInner(void)+0xd [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7298] 
23 000000ff`8c2bfcc0 00007ffd`cd419d87     coreclr!ManagedThreadBase_DispatchMiddle(struct ManagedThreadCallState * pCallState = 0x000000ff`8c2bfd38)+0x85 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7342] 
24 000000ff`8c2bfda0 00007ffd`cd419c79     coreclr!ManagedThreadBase_DispatchOuter(struct ManagedThreadCallState * pCallState = <Value unavailable error>)+0xab [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7501] 
25 (Inline Function) --------`--------     coreclr!ManagedThreadBase_FullTransition(void)+0x2d [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7546] 
26 (Inline Function) --------`--------     coreclr!ManagedThreadBase::KickOff(void)+0x2d [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7581] 
27 000000ff`8c2bfe40 00007ffe`d28226ad     coreclr!ThreadNative::KickOffThread(void * pass = 0x0000029c`8df4e370)+0x79 [D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp @ 230] 
28 000000ff`8c2bfea0 00007ffe`d33ca9f8     KERNEL32!BaseThreadInitThunk+0x1d
29 000000ff`8c2bfed0 00000000`00000000     ntdll!RtlUserThreadStart+0x28

Heap

There is no heap corruption:

0:093> !verifyheap
3,822,041 objects verified, 0 errors.
No heap corruption detected.

RavenDB code

Reference to our code that is shown in the stacktrace:
https://github.com/ravendb/ravendb/blob/9f86bd3d3b08e07f67a5a238bd8dd77933774765/src/Raven.Server/RavenServer.cs#L2491

Author:	arekpalinski
Assignees:	-
Labels:	area-CodeGen-coreclr, untriaged, needs-area-label
Milestone:	-

[AndyAyersMS]

@dotnet/jit-contrib FYI

[AndyAyersMS]

Jitted code leading to the crash is

00007ffd`70f4efb3 48b90000bc6dfd7f0000 mov     rcx, 7FFD6DBC0000h
00007ffd`70f4efbd baffffffff           mov     edx, 0FFFFFFFFh
00007ffd`70f4efc2 e87922505c           call    coreclr!JIT_ClassInitDynamicClass (7ffdcd451240)

Here 7FFD6DBC0000 seems legit, it is the DomainLocalModule for Raven.Server.dll

0:084> !DumpModule /d 00007ffd6da5cf48
Name: D:\workspace\ravendb-6.0\src\Raven.Server\bin\Release\net7.0\Raven.Server.dll
Attributes:              PEFile 
TransientFlags:          00208811 
Assembly:                0000029c82a2b840
BaseAddress:             000002DD373E0000
PEAssembly:              0000029C82A29EF0
ModuleId:                00007FFD6DBC0000

I don't know if the value of FFFFFFFF for the domain ID is normal though... looks like it is used as an index in an array lookup so that value seems like it could be wrong. Presumably the JIT just gets this value from the runtime. So if that's where the problem lies, then either the runtime has given the JIT a bogus value or the JIT has somehow ended up putting the wrong constant in the code stream.

I can't make sense of the DomainLocalAssembly:

0:084> dt -b  (DomainLocalModule*)00007FFD6DBC0000
0x0000029c`82a2bc30
   +0x000 m_pDomainAssembly : 0x0000029c`82a2b840 
   +0x008 m_pDynamicClassTable : VolatilePtr<DomainLocalModule::DynamicClassInfo,DomainLocalModule::DynamicClassInfo *>
      +0x000 m_val            : 0x0000029c`a0cb1da0 
   +0x010 m_aDynamicEntries : Volatile<unsigned __int64>
      +0x000 m_val            : 0x0000029c`82a29ef0
   +0x018 m_pADThunkTable  : VolatilePtr<UMEntryThunk,UMEntryThunk *>
      +0x000 m_val            : 0x00007ffd`6da5cf48 
   +0x020 m_pGCStatics     : 0xbaadf00d`00000000 
   +0x028 m_ModuleIndex    : ModuleIndex
      +0x000 m_dwIndex        : 0
   +0x030 m_pDataBlob      :  "???"
    [00] 0xd8 ''

though the first bit of it looks ok:

0:084> !DumpAssembly 0x0000029c`82a2b840 
Parent Domain:      0000029ca0cb1da0
Name:               D:\workspace\ravendb-6.0\src\Raven.Server\bin\Release\net7.0\Raven.Server.dll
ClassLoader:        0000029C82A2BCE0
  Module
  00007ffd6da5cf48    D:\workspace\ravendb-6.0\src\Raven.Server\bin\Release\net7.0\Raven.Server.dll

@mangod9 can somebody from the runtime take a look too?

[AndyAyersMS]

@EgorBo feel free to dig in from the jit side in the meantime.

[mangod9]

is this a consistent repro? @janvorli in case this is related to assembly unloading or something.

[EgorBo]

@arekpalinski (also cc @redknightlois) have you ever seen a similiar assert in ravendb's unit tests with PGO enabled?

[arekpalinski]

@mangod9 this reproduces quite consistently on our end. It requires a specific setup where 2 nodes of RavenDB cluster are down (A and B) and the client app is talking to one node that is up (node C). The communication is over TCP, and after establishing some connections server C is throwing exceptions because some of them were supposed to be handled by node A or B (we don't do the failover then because we don't have cluster majority at that time - only 1 node is up).

Steps to reproduce:

1. Download: https://drive.google.com/file/d/1C6EKWJAzXLLp5a5aYk9nQodVlFKizKXz/view?usp=sharing
2. Start server C

> Server\C\run.ps1

3. Run Client\Subscription-Client client app

After some time you'll see the crash of server.

You can also reproduce by running the server from the source code in Release mode (debugger can be attached at that time).

1. Checkout v6.0 branch of RavenDB repository (https://github.com/ravendb/ravendb/tree/v6.0)
2. Enable Dynamic PGO (we have disabled it recently)
   https://github.com/ravendb/ravendb/blob/fa782285005641ab1a01dbc862d64cd8c2ac14ac/src/Raven.Server/Raven.Server.csproj#L18
3. Build in Release mode
4. Copy settings.json and RavenData directory from attached ZIP (in C-data directory) to src\Raven.Server\bin\Release\net7.0
5. Start RavenDB server
6. Run the client app

@EgorBo we didn't see anything like that during the unit test runs.

[EgorBo]

@arekpalinski thanks! I was able to reproduce locally