CLR crashing when running RemoteExecutor tests locally on Debug builds

[eiriktsarpalis]

Using the latest commit (007cbcf) on windows-x64, running:

.\build.cmd clr+libs -rc release
dotnet test src/libraries/System.Text.Json/tests/System.Text.Json.SourceGeneration.Tests/System.Text.Json.SourceGeneration.Roslyn4.4.Tests.csproj -f net8.0

Will fail with the following stacktrace:

The active test run was aborted. Reason: Test host process crashed : Fatal error. Internal CLR error. (0x80131506)
   at System.Runtime.Serialization.SerializationGuard.<ThrowIfDeserializationInProgress>g__ThrowIfDeserializationInProgress|0_0(System.Runtime.Serialization.SerializationInfo, System.String, Int32 ByRef)
   at System.Runtime.Serialization.SerializationGuard.ThrowIfDeserializationInProgress(System.String, Int32 ByRef)
   at System.Diagnostics.Process.Start()
   at System.Diagnostics.Process.Start(System.Diagnostics.ProcessStartInfo)
   at Microsoft.DotNet.RemoteExecutor.RemoteExecutor.Invoke(System.Reflection.MethodInfo, System.String[], Microsoft.DotNet.RemoteExecutor.RemoteInvokeOptions, Boolean)
   at Microsoft.DotNet.RemoteExecutor.RemoteExecutor.Invoke(System.Action, Microsoft.DotNet.RemoteExecutor.RemoteInvokeOptions)
   at System.Text.Json.SourceGeneration.Tests.JsonSerializerContextTests.JsonSerializerContext_GeneratedDefault_IsSingleton()
   at System.RuntimeMethodHandle.InvokeMethod(System.Object, Void**, System.Signature, Boolean)
   at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(System.Object, IntPtr*)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(System.Object, System.Reflection.BindingFlags)
   at System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
   at System.Reflection.MethodBase.Invoke(System.Object, System.Object[])
   at Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].CallTestMethod(System.Object)

Which seems to be originating from this call in Process.Start:

runtime/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs

Line 1280 in af651f4

SerializationGuard.ThrowIfDeserializationInProgress("AllowProcessCreation", ref s_cachedSerializationSwitch);

This test has started failing on my local machine in the last few days, but strangely I don't see it impacting CI. Thought I'd file an issue in case others are having the same problem.

Tagging subscribers to this area: @dotnet/area-system-diagnostics-process
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Using the latest commit (007cbcf) on windows-x64, running:

.\build.cmd clr+libs -rc release
dotnet test src/libraries/System.Text.Json/tests/System.Text.Json.SourceGeneration.Tests/System.Text.Json.SourceGeneration.Roslyn4.4.Tests.csproj -f net8.0

Will fail with the following stacktrace:

The active test run was aborted. Reason: Test host process crashed : Fatal error. Internal CLR error. (0x80131506)
   at System.Runtime.Serialization.SerializationGuard.<ThrowIfDeserializationInProgress>g__ThrowIfDeserializationInProgress|0_0(System.Runtime.Serialization.SerializationInfo, System.String, Int32 ByRef)
   at System.Runtime.Serialization.SerializationGuard.ThrowIfDeserializationInProgress(System.String, Int32 ByRef)
   at System.Diagnostics.Process.Start()
   at System.Diagnostics.Process.Start(System.Diagnostics.ProcessStartInfo)
   at Microsoft.DotNet.RemoteExecutor.RemoteExecutor.Invoke(System.Reflection.MethodInfo, System.String[], Microsoft.DotNet.RemoteExecutor.RemoteInvokeOptions, Boolean)
   at Microsoft.DotNet.RemoteExecutor.RemoteExecutor.Invoke(System.Action, Microsoft.DotNet.RemoteExecutor.RemoteInvokeOptions)
   at System.Text.Json.SourceGeneration.Tests.JsonSerializerContextTests.JsonSerializerContext_GeneratedDefault_IsSingleton()
   at System.RuntimeMethodHandle.InvokeMethod(System.Object, Void**, System.Signature, Boolean)
   at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(System.Object, IntPtr*)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(System.Object, System.Reflection.BindingFlags)
   at System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
   at System.Reflection.MethodBase.Invoke(System.Object, System.Object[])
   at Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].CallTestMethod(System.Object)

This test has started failing on my local machine in the last few days, but strangely I don't see it impacting CI. Thought I'd file an issue in case others are having the same problem.

Author:	eiriktsarpalis
Assignees:	-
Labels:	area-System.Diagnostics.Process, untriaged, needs-area-label
Milestone:	-

[stephentoub]

@eiriktsarpalis, you've cleaned and rebuilt from root?

Presumably related to #89706

[eiriktsarpalis]

Yes, the repo was cleaned before building.

[stephentoub]

What does a checked runtime build show? Does it assert?

[eiriktsarpalis]

The active test run was aborted. Reason: Test host process crashed : Assert failure(PID 25160 [0x00006248], Thread: 16784 [0x4190]): Consistency check failed: AV in clr at this callstack:
------
CORECLR! tagCOR_ILMETHOD_TINY::IsTiny + 0x17 (0x00007ff9`6d4f45a7)
CORECLR! DecoderInit + 0x35 (0x00007ff9`6d4f5a35)
CORECLR! `COR_ILMETHOD_DECODER::COR_ILMETHOD_DECODER'::`5'::__Body::Run + 0x65 (0x00007ff9`6d51ca75)
CORECLR! COR_ILMETHOD_DECODER::COR_ILMETHOD_DECODER + 0x82 (0x00007ff9`6d51c702)
CORECLR! ETW::MethodLog::SendMethodJitStartEvent + 0x343 (0x00007ff9`6d258283)
CORECLR! ETW::MethodLog::MethodJitting + 0x37C (0x00007ff9`6d24cb0c)
CORECLR! MethodDesc::JitCompileCodeLockedEventWrapper + 0x465 (0x00007ff9`6cdf7275)
CORECLR! MethodDesc::JitCompileCode + 0x643 (0x00007ff9`6cdf66e3)
CORECLR! MethodDesc::PrepareILBasedCode + 0x691 (0x00007ff9`6cdf9451)
CORECLR! MethodDesc::PrepareCode + 0x20F (0x00007ff9`6cdf8d5f)
CORECLR! CodeVersionManager::PublishVersionableCodeIfNecessary + 0x5A1 (0x00007ff9`6cbcf961)
CORECLR! MethodDesc::DoPrestub + 0x768 (0x00007ff9`6cdede48)
CORECLR! PreStubWorker + 0x47E (0x00007ff9`6cdfee6e)
CORECLR! ThePreStub + 0x55 (0x00007ff9`6d4d85a5)
<no module>! <no symbol> + 0x0 (0x00007ff9`17ee2732)
-----
.AV on tid=0x4190 (16784), cxr=00000018484374C0, exr=00000018484379B0
FAILED: false
CORECLR! CHECK::Trigger + 0x207 (0x00007ff9`6d507bb7)
CORECLR! CLRVectoredExceptionHandlerPhase3 + 0x358 (0x00007ff9`6cc33bd8)
CORECLR! CLRVectoredExceptionHandlerPhase2 + 0xA6 (0x00007ff9`6cc33556)
CORECLR! CLRVectoredExceptionHandler + 0x564 (0x00007ff9`6cc33474)
CORECLR! CLRVectoredExceptionHandlerShim + 0x2A8 (0x00007ff9`6cc33fc8)
NTDLL! RtlGetLengthWithoutLastFullDosOrNtPathElement + 0x3DA (0x00007ff9`e2ba7a9a)
NTDLL! RtlFindCharInUnicodeString + 0x872 (0x00007ff9`e2b4e232)
NTDLL! KiUserExceptionDispatcher + 0x2E (0x00007ff9`e2bd2dee)
CORECLR! tagCOR_ILMETHOD_TINY::IsTiny + 0x17 (0x00007ff9`6d4f45a7)
CORECLR! DecoderInit + 0x35 (0x00007ff9`6d4f5a35)
    File: C:\Users\eitsarpa\devel\dotnet\runtime-feature\src\coreclr\vm\excep.cpp Line: 7119
    Image: C:\Users\eitsarpa\devel\dotnet\runtime-feature\artifacts\bin\testhost\net8.0-windows-Debug-x64\dotnet.exe

[stephentoub]

cc: @jkotas, @AaronRobinsonMSFT (I assume this is to do with UnsafeAccessor)