Distributed transactions: AccessViolationException on arm64

[BruceForstall]

Happens a lot -- see last 30 days:

• 8/25 - 81x failures on net7.0-windows-Release-arm64-CoreCLR_release-Windows.10.Arm64.Open
  • First failure on 8/12 in PR 1939026
  • Impact: 6x failures per day - blocking CI

win-arm64 libraries random JitStress failure

D:\h\w\AFDD09E1\w\B09109AB\e>set COMPlus 
COMPlus_JitStress=36f
COMPlus_TieredCompilation=0

D:\h\w\AFDD09E1\w\B09109AB\e>call RunTests.cmd --runtime-path D:\h\w\AFDD09E1\p 
----- start Thu 08/18/2022  0:42:25.05 ===============  To repro directly: ===================================================== 
pushd D:\h\w\AFDD09E1\w\B09109AB\e\
"D:\h\w\AFDD09E1\p\dotnet.exe" exec --runtimeconfig System.Transactions.Local.Tests.runtimeconfig.json --depsfile System.Transactions.Local.Tests.deps.json xunit.console.dll System.Transactions.Local.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================

D:\h\w\AFDD09E1\w\B09109AB\e>"D:\h\w\AFDD09E1\p\dotnet.exe" exec --runtimeconfig System.Transactions.Local.Tests.runtimeconfig.json --depsfile System.Transactions.Local.Tests.deps.json xunit.console.dll System.Transactions.Local.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Transactions.Local.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Transactions.Local.Tests (found 131 of 142 test cases)
  Starting:    System.Transactions.Local.Tests (parallel test collections = on, max threads = 8)
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
Repeat 2 times:
--------------------------------
   at System.Transactions.DtcProxyShim.DtcInterfaces.IResourceManagerFactory2.CreateEx(System.Guid, System.String, System.Transactions.DtcProxyShim.DtcInterfaces.IResourceManagerSink, System.Guid, System.Object ByRef)
--------------------------------
   at System.Transactions.DtcProxyShim.DtcProxyShimFactory+<>c__DisplayClass10_0.<ConnectToProxy>b__2()
   at System.Transactions.DtcProxyShim.OletxHelper.Retry(System.Action)
   at System.Transactions.DtcProxyShim.DtcProxyShimFactory.ConnectToProxy(System.String, System.Guid, System.Object, Boolean ByRef, Byte[] ByRef, System.Transactions.DtcProxyShim.ResourceManagerShim ByRef)
   at System.Transactions.Oletx.DtcTransactionManager.Initialize()
   at System.Transactions.Oletx.DtcTransactionManager.get_ProxyShimFactory()
   at System.Transactions.Oletx.OletxTransactionManager.CreateTransaction(System.Transactions.TransactionOptions)
   at System.Transactions.TransactionStatePromoted.EnterState(System.Transactions.InternalTransaction)
   at System.Transactions.EnlistableStates.EnlistDurable(System.Transactions.InternalTransaction, System.Guid, System.Transactions.IEnlistmentNotification, System.Transactions.EnlistmentOptions, System.Transactions.Transaction)
   at System.Transactions.Transaction.EnlistDurable(System.Guid, System.Transactions.IEnlistmentNotification, System.Transactions.EnlistmentOptions)
   at System.Transactions.Tests.OleTxTests+OleTxFixture..ctor()
   at System.RuntimeMethodHandle.InvokeMethod(System.Object, Void**, System.Signature, Boolean)
   at System.Reflection.ConstructorInvoker.Invoke(System.Object, IntPtr*, System.Reflection.BindingFlags)
   at System.Reflection.RuntimeConstructorInfo.Invoke(System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
   at Xunit.Sdk.XunitTestClassRunner+<>c__DisplayClass10_0.<CreateClassFixture>b__3()
   at Xunit.Sdk.ExceptionAggregator.Run(System.Action)
   at Xunit.Sdk.XunitTestClassRunner+<CreateClassFixtureAsync>d__11.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestClassRunner+<CreateClassFixtureAsync>d__11, xunit.execution.dotnet, Version=2.4.2.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<CreateClassFixtureAsync>d__11 ByRef)
   at Xunit.Sdk.XunitTestClassRunner.CreateClassFixtureAsync(System.Type)
   at Xunit.Sdk.XunitTestClassRunner+<AfterTestClassStartingAsync>d__13.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestClassRunner+<AfterTestClassStartingAsync>d__13, xunit.execution.dotnet, Version=2.4.2.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<AfterTestClassStartingAsync>d__13 ByRef)
   at Xunit.Sdk.XunitTestClassRunner.AfterTestClassStartingAsync()
   at Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.2.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__37<System.__Canon> ByRef)
   at Xunit.Sdk.TestClassRunner`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunAsync()
   at Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.2.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunTestClassesAsync>d__28<System.__Canon> ByRef)
   at Xunit.Sdk.TestCollectionRunner`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunTestClassesAsync()
   at Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.2.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__27<System.__Canon> ByRef)
   at Xunit.Sdk.TestCollectionRunner`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunAsync()
   at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
   at System.Threading.Tasks.Task.ExecuteEntry()
   at System.Threading.Tasks.SynchronizationContextTaskScheduler+<>c.<.cctor>b__8_0(System.Object)
   at Xunit.Sdk.MaxConcurrencySyncContext.RunOnSyncContext(System.Threading.SendOrPostCallback, System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at Xunit.Sdk.MaxConcurrencySyncContext.WorkerThreadProc()
   at Xunit.Sdk.XunitWorkerThread+<>c.<QueueUserWorkItem>b__5_0(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)

@dotnet/jit-contrib

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

win-arm64 libraries random JitStress failure

D:\h\w\AFDD09E1\w\B09109AB\e>set COMPlus 
COMPlus_JitStress=36f
COMPlus_TieredCompilation=0

D:\h\w\AFDD09E1\w\B09109AB\e>call RunTests.cmd --runtime-path D:\h\w\AFDD09E1\p 
----- start Thu 08/18/2022  0:42:25.05 ===============  To repro directly: ===================================================== 
pushd D:\h\w\AFDD09E1\w\B09109AB\e\
"D:\h\w\AFDD09E1\p\dotnet.exe" exec --runtimeconfig System.Transactions.Local.Tests.runtimeconfig.json --depsfile System.Transactions.Local.Tests.deps.json xunit.console.dll System.Transactions.Local.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================

D:\h\w\AFDD09E1\w\B09109AB\e>"D:\h\w\AFDD09E1\p\dotnet.exe" exec --runtimeconfig System.Transactions.Local.Tests.runtimeconfig.json --depsfile System.Transactions.Local.Tests.deps.json xunit.console.dll System.Transactions.Local.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Transactions.Local.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Transactions.Local.Tests (found 131 of 142 test cases)
  Starting:    System.Transactions.Local.Tests (parallel test collections = on, max threads = 8)
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
Repeat 2 times:
--------------------------------
   at System.Transactions.DtcProxyShim.DtcInterfaces.IResourceManagerFactory2.CreateEx(System.Guid, System.String, System.Transactions.DtcProxyShim.DtcInterfaces.IResourceManagerSink, System.Guid, System.Object ByRef)
--------------------------------
   at System.Transactions.DtcProxyShim.DtcProxyShimFactory+<>c__DisplayClass10_0.<ConnectToProxy>b__2()
   at System.Transactions.DtcProxyShim.OletxHelper.Retry(System.Action)
   at System.Transactions.DtcProxyShim.DtcProxyShimFactory.ConnectToProxy(System.String, System.Guid, System.Object, Boolean ByRef, Byte[] ByRef, System.Transactions.DtcProxyShim.ResourceManagerShim ByRef)
   at System.Transactions.Oletx.DtcTransactionManager.Initialize()
   at System.Transactions.Oletx.DtcTransactionManager.get_ProxyShimFactory()
   at System.Transactions.Oletx.OletxTransactionManager.CreateTransaction(System.Transactions.TransactionOptions)
   at System.Transactions.TransactionStatePromoted.EnterState(System.Transactions.InternalTransaction)
   at System.Transactions.EnlistableStates.EnlistDurable(System.Transactions.InternalTransaction, System.Guid, System.Transactions.IEnlistmentNotification, System.Transactions.EnlistmentOptions, System.Transactions.Transaction)
   at System.Transactions.Transaction.EnlistDurable(System.Guid, System.Transactions.IEnlistmentNotification, System.Transactions.EnlistmentOptions)
   at System.Transactions.Tests.OleTxTests+OleTxFixture..ctor()
   at System.RuntimeMethodHandle.InvokeMethod(System.Object, Void**, System.Signature, Boolean)
   at System.Reflection.ConstructorInvoker.Invoke(System.Object, IntPtr*, System.Reflection.BindingFlags)
   at System.Reflection.RuntimeConstructorInfo.Invoke(System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
   at Xunit.Sdk.XunitTestClassRunner+<>c__DisplayClass10_0.<CreateClassFixture>b__3()
   at Xunit.Sdk.ExceptionAggregator.Run(System.Action)
   at Xunit.Sdk.XunitTestClassRunner+<CreateClassFixtureAsync>d__11.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestClassRunner+<CreateClassFixtureAsync>d__11, xunit.execution.dotnet, Version=2.4.2.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<CreateClassFixtureAsync>d__11 ByRef)
   at Xunit.Sdk.XunitTestClassRunner.CreateClassFixtureAsync(System.Type)
   at Xunit.Sdk.XunitTestClassRunner+<AfterTestClassStartingAsync>d__13.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestClassRunner+<AfterTestClassStartingAsync>d__13, xunit.execution.dotnet, Version=2.4.2.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<AfterTestClassStartingAsync>d__13 ByRef)
   at Xunit.Sdk.XunitTestClassRunner.AfterTestClassStartingAsync()
   at Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.2.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__37<System.__Canon> ByRef)
   at Xunit.Sdk.TestClassRunner`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunAsync()
   at Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.2.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunTestClassesAsync>d__28<System.__Canon> ByRef)
   at Xunit.Sdk.TestCollectionRunner`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunTestClassesAsync()
   at Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.4.2.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__27<System.__Canon> ByRef)
   at Xunit.Sdk.TestCollectionRunner`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunAsync()
   at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
   at System.Threading.Tasks.Task.ExecuteEntry()
   at System.Threading.Tasks.SynchronizationContextTaskScheduler+<>c.<.cctor>b__8_0(System.Object)
   at Xunit.Sdk.MaxConcurrencySyncContext.RunOnSyncContext(System.Threading.SendOrPostCallback, System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at Xunit.Sdk.MaxConcurrencySyncContext.WorkerThreadProc()
   at Xunit.Sdk.XunitWorkerThread+<>c.<QueueUserWorkItem>b__5_0(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)

@dotnet/jit-contrib

Author:	BruceForstall
Assignees:	-
Labels:	arch-arm64, os-windows, JitStress, area-CodeGen-coreclr
Milestone:	7.0.0

[kunalspathak]

Could this be related to recent changes in this area? I am able to repro this even without the jitstress flags.

This was recently ported to Windows in #72051 and seems related to that.

Here is the call stack on windows/arm64

00 MSDTCPRX!CIResourceManagerFactory::CreateEx
01 xunit_console!ILStubClass.IL_STUB_CLRtoCOM(System.Guid, System.String, System.Transactions.DtcProxyShim.DtcInterfaces.IResourceManagerSink, System.Guid, System.Object ByRef)
02 System_Transactions_Local!System.Transactions.DtcProxyShim.DtcProxyShimFactory+<>c__DisplayClass10_0.<ConnectToProxy>b__2()
03 System_Transactions_Local!System.Transactions.DtcProxyShim.OletxHelper.Retry(System.Action)
04 System_Transactions_Local!System.Transactions.DtcProxyShim.DtcProxyShimFactory.ConnectToProxy(System.String, System.Guid, System.Object, Boolean ByRef, Byte[] ByRef, System.Transactions.DtcProxyShim.ResourceManagerShim ByRef)
05 System_Transactions_Local!System.Transactions.Oletx.DtcTransactionManager.Initialize()
06 System_Transactions_Local!System.Transactions.Oletx.DtcTransactionManager.get_ProxyShimFactory()
07 System_Transactions_Local!System.Transactions.Oletx.OletxTransactionManager.CreateTransaction(System.Transactions.TransactionOptions)
08 System_Transactions_Local!System.Transactions.TransactionStatePromoted.EnterState(System.Transactions.InternalTransaction)
09 System_Transactions_Local!System.Transactions.EnlistableStates.EnlistDurable(System.Transactions.InternalTransaction, System.Guid, System.Transactions.IEnlistmentNotification, System.Transactions.EnlistmentOptions, System.Transactions.Transaction)
0a System_Transactions_Local!System.Transactions.Transaction.EnlistDurable(System.Guid, System.Transactions.IEnlistmentNotification, System.Transactions.EnlistmentOptions)
0b System_Transactions_Local_Tests!System.Transactions.Tests.OleTxTests+OleTxFixture..ctor()
0c coreclr!CallDescrWorkerInternal

[roji]

Will look at this ASAP

[roji]

/cc @AaronRobinsonMSFT

[kunalspathak]

I would be curious to know why we don't see these failures in other CI pipelines.