Dataflow analysis lockup compiling System.Runtime.Serialization.Formatters.Tests

[MichalStrehovsky]

It's possible this is an illinker bug, but I'm not ready to spend time extracting an IL Linker repro:

To repro, build the repo how you build it (e.g. build.cmd clr.aot+libs -rc Debug -lc Release). Then:

pushd src\libraries\System.Runtime.Serialization.Formatters\tests
dotnet build -c Release /p:TestNativeAot=true /p:UseNativeAotCoreLib=true /p:RuntimeConfiguration=Debug /p:LibrariesConfiguration=Release /p:TargetFramework=net7.0-windows

Once it tells you it's generating native code, taskkill ilc.exe and find the response file to invoke ilc.exe under artifacts\obj\System.Runtime.Serialization.Formatters.Tests\Release\net7.0-windows\native. Open src\coreclr\tools\aot\ilc.sln in VS, set configuration to the one you built (Debug in my case from the -rc Debug argument), and set the response file as the command line argument. You might want to append --parallelism:1 to make things not multithreaded.

The method we're stuck on is [System.Runtime.Serialization.Formatters.Tests]System.Runtime.Serialization.Formatters.Tests.BinaryFormatterTests.SerializableObjects()

Example callstack:

 	System.Private.CoreLib.dll!System.RuntimeType.GetFieldCandidates(string name, System.Reflection.BindingFlags bindingAttr, bool allowPrefixLookup)	Unknown
 	System.Private.CoreLib.dll!System.RuntimeType.GetFields(System.Reflection.BindingFlags bindingAttr)	Unknown
 	System.Private.CoreLib.dll!System.ValueType.Equals(object obj)	Unknown
 	System.Private.CoreLib.dll!System.Collections.Generic.HashSet<System.Collections.Generic.KeyValuePair<int, ILCompiler.Dataflow.ValueBasicBlockPair>>.Remove(System.Collections.Generic.KeyValuePair<int, ILCompiler.Dataflow.ValueBasicBlockPair> item)	Unknown
 	System.Private.CoreLib.dll!System.Collections.Generic.HashSet<System.Collections.Generic.KeyValuePair<int, ILCompiler.Dataflow.ValueBasicBlockPair>>.ExceptWith(System.Collections.Generic.IEnumerable<System.Collections.Generic.KeyValuePair<int, ILCompiler.Dataflow.ValueBasicBlockPair>> other)	Unknown
>	ILCompiler.Compiler.dll!ILLink.Shared.TrimAnalysis.ArrayValue.Equals(ILLink.Shared.TrimAnalysis.ArrayValue otherArr) Line 79	C#
 	System.Private.CoreLib.dll!System.Collections.Generic.GenericEqualityComparer<ILLink.Shared.DataFlow.SingleValue>.Equals(ILLink.Shared.DataFlow.SingleValue x, ILLink.Shared.DataFlow.SingleValue y)	Unknown
 	System.Private.CoreLib.dll!System.Collections.Generic.HashSet<ILLink.Shared.DataFlow.SingleValue>.AddIfNotPresent(ILLink.Shared.DataFlow.SingleValue value, out int location)	Unknown
 	System.Private.CoreLib.dll!System.Collections.Generic.HashSet<ILLink.Shared.DataFlow.SingleValue>.UnionWith(System.Collections.Generic.IEnumerable<ILLink.Shared.DataFlow.SingleValue> other)	Unknown
 	System.Private.CoreLib.dll!System.Collections.Generic.HashSet<ILLink.Shared.DataFlow.SingleValue>.HashSet(System.Collections.Generic.IEnumerable<ILLink.Shared.DataFlow.SingleValue> collection, System.Collections.Generic.IEqualityComparer<ILLink.Shared.DataFlow.SingleValue> comparer)	Unknown
 	ILCompiler.Compiler.dll!ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>.Clone() Line 190	C#
 	ILCompiler.Compiler.dll!ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue>.Meet(ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue> left, ILLink.Shared.DataFlow.ValueSet<ILLink.Shared.DataFlow.SingleValue> right) Line 159	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.MethodBodyScanner.ScanStfld(Internal.IL.MethodIL methodBody, int offset, Internal.IL.ILOpcode opcode, Internal.TypeSystem.FieldDesc field, System.Collections.Generic.Stack<ILCompiler.Dataflow.StackSlot> currentStack, ref ILCompiler.Dataflow.InterproceduralState interproceduralState) Line 1134	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.MethodBodyScanner.Scan(Internal.IL.MethodIL methodBody, ref ILCompiler.Dataflow.InterproceduralState interproceduralState) Line 664	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.ReflectionMethodBodyScanner.Scan(Internal.IL.MethodIL methodBody, ref ILCompiler.Dataflow.InterproceduralState interproceduralState) Line 106	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.MethodBodyScanner.InterproceduralScan(Internal.IL.MethodIL startingMethodBody) Line 317	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.ReflectionMethodBodyScanner.InterproceduralScan(Internal.IL.MethodIL methodBody) Line 97	C#
 	ILCompiler.Compiler.dll!ILCompiler.Dataflow.ReflectionMethodBodyScanner.ScanAndProcessReturnValue(ILCompiler.DependencyAnalysis.NodeFactory factory, ILLink.Shared.TrimAnalysis.FlowAnnotations annotations, ILCompiler.Logger logger, Internal.IL.MethodIL methodBody) Line 121	C#
 	ILCompiler.Compiler.dll!ILCompiler.DependencyAnalysis.DataflowAnalyzedMethodNode.GetStaticDependencies(ILCompiler.DependencyAnalysis.NodeFactory factory) Line 36	C#

Cc @vitek-karas @sbomer

[vitek-karas]

I must be doing something wrong - the above repro steps don't hang. ILC successfully process the input in about 20 seconds (maybe a tiny bit more). This is on very recent main branch - no local changes.

[MichalStrehovsky]

Tried enabling the test in #73104.

The CI has been locked up like this for 20 minutes.

  System.Runtime.Serialization.Formatters.Tests -> /__w/1/s/artifacts/bin/System.Runtime.Serialization.Formatters.Tests/Release/net7.0-linux/System.Runtime.Serialization.Formatters.Tests.dll
  Optimizing assemblies for size may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink
  Generating compatible native code. To optimize for size or speed, visit https://aka.ms/OptimizeNativeAOT

It's possible the repro steps don't repro it. I'm still trying to figure out what is the right way to dotnet build a single test. What I wrote above has worked for me in the past but maybe it requires some previous artifacts leftover from a previous build attempt. Not sure.

What I always do locally is I change the definition of SmokeTestProject in src/libraries/tests.proj to only have the test of interest to me and run build.cmd libs.tests -rc Debug -lc Release /p:TestNativeAot=true /p:RunSmokeTestsOnly=true /p:UseNativeAotCoreLib=true. It's not nice but it works :/

[MichalStrehovsky]

It's probably not a lockup - I just don't have enough patience. Extracted a repro to dotnet/linker#2977.