Test failure readytorun/determinism/crossgen2determinism/crossgen2determinism.sh

[VincentBu]

Run: runtime 20210817.93

Failed test:

CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open

- readytorun/determinism/crossgen2determinism/crossgen2determinism.sh
- readytorun/tests/mainv2/mainv2.sh
- readytorun/coreroot_determinism/coreroot_determinism/coreroot_determinism.sh
- baseservices/TieredCompilation/BasicTestWithMcj/BasicTestWithMcj.sh
- readytorun/tests/mainv1/mainv1.sh

CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open

- readytorun/determinism/crossgen2determinism/crossgen2determinism.sh
- readytorun/tests/mainv2/mainv2.sh
- readytorun/coreroot_determinism/coreroot_determinism/coreroot_determinism.sh
- baseservices/TieredCompilation/BasicTestWithMcj/BasicTestWithMcj.sh
- readytorun/tests/mainv1/mainv1.sh

CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open

- readytorun/determinism/crossgen2determinism/crossgen2determinism.sh
- baseservices/TieredCompilation/BasicTestWithMcj/BasicTestWithMcj.sh
- readytorun/coreroot_determinism/coreroot_determinism/coreroot_determinism.sh
- readytorun/tests/mainv2/mainv2.sh
- readytorun/tests/mainv1/mainv1.sh

One of error message:

Unhandled exception. System.AggregateException: One or more errors occurred. (Code generation failed for method '[crossgen2smoke]Program.TestSES_NestedR8(SingleElementStruct_NestedR8,object)') (Code generation failed for method '[crossgen2smoke]Program.TestSES_NestedU8(SingleElementStruct_NestedU8,object)')
---> ILCompiler.CodeGenerationFailedException: Code generation failed for method '[crossgen2smoke]Program.TestSES_NestedR8(SingleElementStruct_NestedR8,object)'
---> System.MissingMethodException: Method not found: 'Void System.Diagnostics.Debug.Assert(Boolean, AssertInterpolatedStringHandler)'.
at Internal.JitInterface.SystemVStructClassificator.TypeDef2SystemVClassification(TypeDesc typeDesc)
at Internal.JitInterface.SystemVStructClassificator.GetSystemVAmd64PassStructInRegisterDescriptor(TypeDesc typeDesc, SYSTEMV_AMD64_CORINFO_STRUCT_REG_PASSING_DESCRIPTOR& structPassInRegDescPtr)
at Internal.JitInterface.CorInfoImpl.getSystemVAmd64PassStructInRegisterDescriptor(IntPtr thisHandle, IntPtr* ppException, CORINFO_CLASS_STRUCT* structHnd, SYSTEMV_AMD64_CORINFO_STRUCT_REG_PASSING_DESCRIPTOR* structPassInRegDescPtr)
--- End of inner exception stack trace ---
at Internal.JitInterface.CorInfoImpl.CompileMethodInternal(IMethodNode methodCodeNodeNeedingCode, MethodIL methodIL)
at Internal.JitInterface.CorInfoImpl.CompileMethod(MethodWithGCInfo methodCodeNodeNeedingCode, Logger logger)
at ILCompiler.ReadyToRunCodegenCompilation.<ComputeDependencyNodeDependencies>b__36_0(DependencyNodeCore1 dependency) at System.Threading.Tasks.Parallel.<>c__DisplayClass19_01.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion) in //src/libraries/System.Threading.Tasks.Parallel/src/System/Threading/Tasks/Parallel.cs:line 1147
--- End of stack trace from previous location ---
at System.Threading.Tasks.Parallel.<>c__DisplayClass19_01.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion) in /_/src/libraries/System.Threading.Tasks.Parallel/src/System/Threading/Tasks/Parallel.cs:line 1190 at System.Threading.Tasks.TaskReplicator.Replica.Execute() in /_/src/libraries/System.Threading.Tasks.Parallel/src/System/Threading/Tasks/TaskReplicator.cs:line 77 --- End of inner exception stack trace --- at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction1 action, ParallelOptions options, Boolean stopOnFirstFailure) in //src/libraries/System.Threading.Tasks.Parallel/src/System/Threading/Tasks/TaskReplicator.cs:line 155
at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Func4 bodyWithLocal, Func1 localInit, Action1 localFinally) in /_/src/libraries/System.Threading.Tasks.Parallel/src/System/Threading/Tasks/Parallel.cs:line 1079 --- End of stack trace from previous location --- at System.Threading.Tasks.Parallel.ThrowSingleCancellationExceptionOrOtherException(ICollection exceptions, CancellationToken cancelToken, Exception otherException) in /_/src/libraries/System.Threading.Tasks.Parallel/src/System/Threading/Tasks/Parallel.cs:line 3409 at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Func4 bodyWithLocal, Func1 localInit, Action1 localFinally) in //src/libraries/System.Threading.Tasks.Parallel/src/System/Threading/Tasks/Parallel.cs:line 1225
at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable1 source, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Action3 bodyWithStateAndIndex, Func4 bodyWithStateAndLocal, Func5 bodyWithEverything, Func1 localInit, Action1 localFinally) in //src/libraries/System.Threading.Tasks.Parallel/src/System/Threading/Tasks/Parallel.cs:line 2110
at System.Th


Stack trace
   at readytorun_determinism._crossgen2determinism_crossgen2determinism_._crossgen2determinism_crossgen2determinism_sh()

Runfo Tracking Issue: readytorun/determinism/crossgen2determinism/crossgen2determinism.sh

Build	Definition	Kind	Run Name	Console	Core Dump	Test Results	Run Client
1301452	runtime	PR 57483	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1301452	runtime	PR 57483	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1301452	runtime	PR 57483	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1301321	runtime	PR 57637	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1301321	runtime	PR 57637	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1301321	runtime	PR 57637	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1301250	runtime	PR 57605	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1301250	runtime	PR 57605	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1301250	runtime	PR 57605	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1301070	runtime	PR 57585	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1301070	runtime	PR 57585	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1301070	runtime	PR 57585	CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1301070	runtime	PR 57585	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1301031	runtime	Rolling	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1301031	runtime	Rolling	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1301031	runtime	Rolling	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1300928	runtime	PR 57303	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300928	runtime	PR 57303	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300928	runtime	PR 57303	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1300925	runtime	Rolling	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300925	runtime	Rolling	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300925	runtime	Rolling	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1300829	runtime	PR 57508	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300829	runtime	PR 57508	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300829	runtime	PR 57508	CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1300829	runtime	PR 57508	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1300704	runtime	PR 57244	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300704	runtime	PR 57244	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300704	runtime	PR 57244	CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1300704	runtime	PR 57244	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1300557	runtime	Rolling	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300557	runtime	Rolling	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300557	runtime	Rolling	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1300343	runtime	PR 57303	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300343	runtime	PR 57303	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300343	runtime	PR 57303	CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1300343	runtime	PR 57303	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1300342	runtime	Rolling	CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1300342	runtime	Rolling	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300342	runtime	Rolling	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300194	runtime	PR 57616	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300194	runtime	PR 57616	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300194	runtime	PR 57616	CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1300194	runtime	PR 57616	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1300142	runtime	PR 57508	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300142	runtime	PR 57508	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300142	runtime	PR 57508	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1300137	runtime	PR 57615	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300137	runtime	PR 57615	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300137	runtime	PR 57615	CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1300137	runtime	PR 57615	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1300088	runtime	PR 57614	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1300088	runtime	PR 57614	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299991	runtime	PR 57101	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299989	runtime	PR 52402	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299989	runtime	PR 52402	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299989	runtime	PR 52402	CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299989	runtime	PR 52402	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299961	runtime	PR 57613	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299961	runtime	PR 57613	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299946	runtime	PR 57354	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299946	runtime	PR 57354	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299946	runtime	PR 57354	CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299946	runtime	PR 57354	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299890	runtime	PR 57396	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299890	runtime	PR 57396	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299890	runtime	PR 57396	CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299890	runtime	PR 57396	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299793	runtime	PR 57607	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299793	runtime	PR 57607	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299793	runtime	PR 57607	CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299793	runtime	PR 57607	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299793	runtime	PR 57607	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299793	runtime	PR 57607	CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299793	runtime	PR 57607	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299767	runtime	Rolling	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299767	runtime	Rolling	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299767	runtime	Rolling	CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299767	runtime	Rolling	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299737	runtime	PR 57605	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299737	runtime	PR 57605	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299737	runtime	PR 57605	CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299737	runtime	PR 57605	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299724	runtime	Rolling	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299724	runtime	Rolling	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299724	runtime	Rolling	CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299724	runtime	Rolling	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299722	runtime	PR 57602	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299722	runtime	PR 57602	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299722	runtime	PR 57602	CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299722	runtime	PR 57602	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299702	runtime	PR 56501	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299702	runtime	PR 56501	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299702	runtime	PR 56501	CoreCLR OSX x64 Checked @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299702	runtime	PR 56501	CoreCLR OSX x64 Checked no_tiered_compilation @ OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1299656	runtime	PR 57603	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299656	runtime	PR 57603	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299641	runtime	PR 56659	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299641	runtime	PR 56659	CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1299582	runtime	PR 57023	CoreCLR Linux x64 Checked @ Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py

Displaying 100 of 123 results

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
32	34	34

[hoyosjs]

Turning this into a tracking issue. This is currently one of the big hits.

[hoyosjs]

This slipped to release/6.0: https://runfo.azurewebsites.net/search/tests/?q=started%3A~7%20definition%3Aruntime%20targetbranch%3Arelease%2F6.0%20name%3A%22readytorun%2Fdeterminism%2Fcrossgen2determinism%2Fcrossgen2determinism.sh%22

it always complains that Void System.Diagnostics.Debug.Assert(Boolean, AssertInterpolatedStringHandler) could not be found, which makes sense as the method was modified in https://github.com/dotnet/runtime/pull/57536/files to take ref AssertInterpolatedStringHandler cc: @dotnet/crossgen-contrib @stephentoub

[stephentoub]

My bad; I looked at the failures before merging that PR, but didn't comprehend that a few of them were real... the problem with so many tests failing is sometimes real signal gets hidden by the noise.

That said, I don't know what the problem is, or rather, why it's happening. The signature changed from an argument being non-ref to ref, and apparently the code is built expecting ref but then is running against something without the ref. Is crossgen2 somehow built against live refs but run against an older corelib?

[jkotas]

crossgen2 it is built against live refs, but it runs against the tools runtime in some cases. I think the quick workaround is to just comment out the one place in crossgen2 where the string interpolating assert is used (or replace it with regular assert).

I expect we will hit the same problem again for .NET 7 version change (the loader version sanity check will fail on 7.0 assemblies running on 6.0 runtime), so we will likely need to implement proper fix soon.

[stephentoub]

I think the quick workaround is to just comment out the one place in crossgen2 where the string interpolating assert is used

On it...