ILC fails with System.OverflowException: Arithmetic operation resulted in an overflow.

[rolfbjarne]

Description

ILC fails with System.OverflowException: Arithmetic operation resulted in an overflow.

A lot of memory is also used (at one time over 30gb on my machine).

Reproduction Steps

Repro created by passing --make-repro-path to ilc:
274257939_monotouchtest.zip

Expected behavior

Successful ILC compilation.

Actual behavior

There's a lot of output, zipped: ilcoutput.zip

Here's an extract:

ILC: Method '[System.Linq.Parallel]System.Linq.Parallel.GroupJoinQueryOperator`4<System.__Canon,System.__Canon,System.__Canon,System.__Canon>.WrapPartitionedStreamHelper<Pair`2<Pair`2<Pair`2<Pair`2<__Canon,__Canon>,__Canon>,__Canon>,__Canon>,Pair`2<bool,ConcatKey`2<Pair`2<__Canon,__Canon>,Pair`2<int32,int32>>>>(PartitionedStream`2<Pair`2<__Canon,__Canon>,Pair`2<Pair`2<Pair`2<Pair`2<__Canon,__Canon>,__Canon>,__Canon>,__Canon>>,HashLookupBuilder`3<IEnumerable`1<__Canon>,Pair`2<bool,ConcatKey`2<Pair`2<__Canon,__Canon>,Pair`2<int32,int32>>>,__Canon>[],IComparer`1<Pair`2<bool,ConcatKey`2<Pair`2<__Canon,__Canon>,Pair`2<int32,int32>>>>,IPartitionedStreamRecipient`1<__Canon>,int32,CancellationToken)' will always throw because: Failed to load type 'System.Linq.Parallel.GroupJoinQueryOperator`4<System.__Canon, System.__Canon, System.__Canon, System.__Canon>' from assembly 'System.Linq.Parallel, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
Error: One or more errors occurred. (Arithmetic operation resulted in an overflow.) (Arithmetic operation resulted in an overflow.) (Code generation failed for method '[System.Linq.Parallel]System.Linq.Parallel.BinaryQueryOperator`3+BinaryQueryOperatorResults+RightChildResultsRecipient`1<System.__Canon,System.__Canon,System.__Canon,System.Linq.Parallel.Pair`2<System.Linq.Parallel.Pair`2<System.__Canon,int32>,System.Linq.Parallel.Pair`2<System.__Canon,System.Linq.Parallel.ConcatKey`2<System.__Canon,System.__Canon>>>>.Receive<Pair`2<Pair`2<__Canon,Pair`2<int32,int32>>,ConcatKey`2<Pair`2<__Canon,int32>,__Canon>>>(PartitionedStream`2<__Canon,Pair`2<Pair`2<__Canon,Pair`2<int32,int32>>,ConcatKey`2<Pair`2<__Canon,int32>,__Canon>>>)') (Code generation failed for method '[System.Linq.Parallel]System.Linq.Parallel.GroupJoinQueryOperator`4<System.__Canon,System.__Canon,System.__Canon,System.__Canon>.WrapPartitionedStream<Pair`2<ConcatKey`2<int32,__Canon>,Pair`2<__Canon,Pair`2<bool,__Canon>>>,Pair`2<Pair`2<int32,Pair`2<bool,__Canon>>,Pair`2<__Canon,int32>>>(PartitionedStream`2<__Canon,Pair`2<ConcatKey`2<int32,__Canon>,Pair`2<__Canon,Pair`2<bool,__Canon>>>>,PartitionedStream`2<__Canon,Pair`2<Pair`2<int32,Pair`2<bool,__Canon>>,Pair`2<__Canon,int32>>>,IPartitionedStreamRecipient`1<__Canon>,bool,QuerySettings)') (Code generation failed for method '[System.Linq.Parallel]System.Linq.Parallel.BinaryQueryOperator`3+BinaryQueryOperatorResults+RightChildResultsRecipient`1<System.__Canon,System.__Canon,System.__Canon,System.Linq.Parallel.Pair`2<System.Linq.Parallel.ConcatKey`2<int32,System.Linq.Parallel.Pair`2<System.__Canon,int32>>,int32>>.Receive<Pair`2<Pair`2<__Canon,Pair`2<bool,__Canon>>,Pair`2<Pair`2<__Canon,int32>,ConcatKey`2<__Canon,__Canon>>>>(PartitionedStream`2<__Canon,Pair`2<Pair`2<__Canon,Pair`2<bool,__Canon>>,Pair`2<Pair`2<__Canon,int32>,ConcatKey`2<__Canon,__Canon>>>>)') (Code generation failed for method '[System.Linq.Parallel]System.Linq.Parallel.BinaryQueryOperator`3+BinaryQueryOperatorResults+RightChildResultsRecipient`1<System.__Canon,System.__Canon,System.__Canon,System.Linq.Parallel.Pair`2<System.Linq.Parallel.Pair`2<System.Linq.Parallel.ConcatKey`2<System.__Canon,System.__Canon>,int32>,System.Linq.Parallel.Pair`2<System.__Canon,int32>>>.Receive<ConcatKey`2<Pair`2<int32,Pair`2<bool,__Canon>>,Pair`2<Pair`2<__Canon,Pair`2<bool,__Canon>>,int32>>>(PartitionedStream`2<__Canon,ConcatKey`2<Pair`2<int32,Pair`2<bool,__Canon>>,Pair`2<Pair`2<__Canon,Pair`2<bool,__Canon>>,int32>>>)') (Code generation failed for method '[System.Linq.Parallel]System.Linq.Parallel.BinaryQueryOperator`3+BinaryQueryOperatorResults+RightChildResultsRecipient`1<System.__Canon,System.__Canon,System.__Canon,System.Linq.Parallel.Pair`2<System.Linq.Parallel.Pair`2<int32,int32>,System.Linq.Parallel.Pair`2<System.Linq.Parallel.Pair`2<int32,int32>,int32>>>.Receive<Pair`2<Pair`2<int32,__Canon>,ConcatKey`2<__Canon,ConcatKey`2<__Canon,int32>>>>(PartitionedStream`2<__Canon,Pair`2<Pair`2<int32,__Canon>,ConcatKey`2<__Canon,ConcatKey`2<__Canon,int32>>>>)') (Code generation failed for method '[System.Linq.Parallel]System.Linq.Parallel.BinaryQueryOperator`3+BinaryQueryOperatorResults+RightChildResultsRecipient`1<System.__Canon,System.__Canon,System.__Canon,System.Linq.Parallel.ConcatKey`2<System.__Canon,System.Linq.Parallel.Pair`2<System.Linq.Parallel.ConcatKey`2<System.__Canon,System.__Canon>,int32>>>.Receive<Pair`2<ConcatKey`2<__Canon,Pair`2<int32,int32>>,Pair`2<Pair`2<__Canon,int32>,Pair`2<int32,int32>>>>(PartitionedStream`2<__Canon,Pair`2<ConcatKey`2<__Canon,Pair`2<int32,int32>>,Pair`2<Pair`2<__Canon,int32>,Pair`2<int32,int32>>>>)') (Code generation failed for method '[System.Linq.Parallel]System.Linq.Parallel.BinaryQueryOperator`3+BinaryQueryOperatorResults+RightChildResultsRecipient`1<System.__Canon,System.__Canon,System.__Canon,System.Linq.Parallel.ConcatKey`2<int32,System.Linq.Parallel.Pair`2<int32,System.Linq.Parallel.Pair`2<bool,System.__Canon>>>>.Receive<ConcatKey`2<Pair`2<__Canon,Pair`2<__Canon,int32>>,ConcatKey`2<int32,ConcatKey`2<__Canon,__Canon>>>>(PartitionedStream`2<__Canon,ConcatKey`2<Pair`2<__Canon,Pair`2<__Canon,int32>>,ConcatKey`2<int32,ConcatKey`2<__Canon,__Canon>>>>)') (Code generation failed for method '[System.Linq.Parallel]System.Linq.Parallel.BinaryQueryOperator`3+BinaryQueryOperatorResults+RightChildResultsRecipient`1<System.__Canon,System.__Canon,System.__Canon,System.Linq.Parallel.Pair`2<System.Linq.Parallel.Pair`2<System.__Canon,System.Linq.Parallel.Pair`2<bool,System.__Canon>>,System.Linq.Parallel.ConcatKey`2<int32,System.Linq.Parallel.Pair`2<int32,int32>>>>.Receive<Pair`2<ConcatKey`2<int32,__Canon>,Pair`2<bool,__Canon>>>(PartitionedStream`2<__Canon,Pair`2<ConcatKey`2<int32,__Canon>,Pair`2<bool,__Canon>>>)') (Arithmetic operation resulted in an overflow.) (Code generation failed for method '[System.Linq.Parallel]System.Linq.Parallel.UnionQueryOperator`1<System.__Canon>.WrapPartitionedStreamFixedBothTypes<Pair`2<int32,Pair`2<bool,Pair`2<int32,int32>>>,Pair`2<ConcatKey`2<__Canon,int32>,Pair`2<__Canon,int32>>>(PartitionedStream`2<Pair`2<__Canon,NoKeyMemoizationRequired>,Pair`2<int32,Pair`2<bool,Pair`2<int32,int32>>>>,PartitionedStream`2<Pair`2<__Canon,NoKeyMemoizationRequired>,Pair`2<ConcatKey`2<__Canon,int32>,Pair`2<__Canon,int32>>>,IPartitionedStreamRecipient`1<__Canon>,int32,CancellationToken)')
System.AggregateException: One or more errors occurred. (Arithmetic operation resulted in an overflow.) (Arithmetic operation resulted in an overflow.) (Code generation failed for method '[System.Linq.Parallel]System.Linq.Parallel.BinaryQueryOperator`3+BinaryQueryOperatorResults+RightChildResultsRecipient`1<System.__Canon,System.__Canon,System.__Canon,System.Linq.Parallel.Pair`2<System.Linq.Parallel.Pair`2<System.__Canon,int32>,System.Linq.Parallel.Pair`2<System.__Canon,System.Linq.Parallel.ConcatKey`2<System.__Canon,System.__Canon>>>>.Receive<Pair`2<Pair`2<__Canon,Pair`2<int32,int32>>,ConcatKey`2<Pair`2<__Canon,int32>,__Canon>>>(PartitionedStream`2<__Canon,Pair`2<Pair`2<__Canon,Pair`2<int32,int32>>,ConcatKey`2<Pair`2<__Canon,int32>,__Canon>>>)') (Code generation failed for method '[System.Linq.Parallel]System.Linq.Parallel.GroupJoinQueryOperator`4<System.__Canon,System.__Canon,System.__Canon,System.__Canon>.WrapPartitionedStream<Pair`2<ConcatKey`2<int32,__Canon>,Pair`2<__Canon,Pair`2<bool,__Canon>>>,Pair`2<Pair`2<int32,Pair`2<bool,__Canon>>,Pair`2<__Canon,int32>>>(PartitionedStream`2<__Canon,Pair`2<ConcatKey`2<int32,__Canon>,Pair`2<__Canon,Pair`2<bool,__Canon>>>>,PartitionedStream`2<__Canon,Pair`2<Pair`2<int32,Pair`2<bool,__Canon>>,Pair`2<__Canon,int32>>>,IPartitionedStreamRecipient`1<__Canon>,bool,QuerySettings)') (Code generation failed for method '[System.Linq.Parallel]System.Linq.Parallel.BinaryQueryOperator`3+BinaryQueryOperatorResults+RightChildResultsRecipient`1<System.__Canon,System.__Canon,System.__Canon,System.Linq.Parallel.Pair`2<System.Linq.Parallel.ConcatKey`2<int32,System.Linq.Parallel.Pair`2<System.__Canon,int32>>,int32>>.Receive<Pair`2<Pair`2<__Canon,Pair`2<bool,__Canon>>,Pair`2<Pair`2<__Canon,int32>,ConcatKey`2<__Canon,__Canon>>>>(PartitionedStream`2<__Canon,Pair`2<Pair`2<__Canon,Pair`2<bool,__Canon>>,Pair`2<Pair`2<__Canon,int32>,ConcatKey`2<__Canon,__Canon>>>>)') (Code generation failed for method '[System.Linq.Parallel]System.Linq.Parallel.BinaryQueryOperator`3+BinaryQueryOperatorResults+RightChildResultsRecipient`1<System.__Canon,System.__Canon,System.__Canon,System.Linq.Parallel.Pair`2<System.Linq.Parallel.Pair`2<System.Linq.Parallel.ConcatKey`2<System.__Canon,System.__Canon>,int32>,System.Linq.Parallel.Pair`2<System.__Canon,int32>>>.Receive<ConcatKey`2<Pair`2<int32,Pair`2<bool,__Canon>>,Pair`2<Pair`2<__Canon,Pair`2<bool,__Canon>>,int32>>>(PartitionedStream`2<__Canon,ConcatKey`2<Pair`2<int32,Pair`2<bool,__Canon>>,Pair`2<Pair`2<__Canon,Pair`2<bool,__Canon>>,int32>>>)') (Code generation failed for method '[System.Linq.Parallel]System.Linq.Parallel.BinaryQueryOperator`3+BinaryQueryOperatorResults+RightChildResultsRecipient`1<System.__Canon,System.__Canon,System.__Canon,System.Linq.Parallel.Pair`2<System.Linq.Parallel.Pair`2<int32,int32>,System.Linq.Parallel.Pair`2<System.Linq.Parallel.Pair`2<int32,int32>,int32>>>.Receive<Pair`2<Pair`2<int32,__Canon>,ConcatKey`2<__Canon,ConcatKey`2<__Canon,int32>>>>(PartitionedStream`2<__Canon,Pair`2<Pair`2<int32,__Canon>,ConcatKey`2<__Canon,ConcatKey`2<__Canon,int32>>>>)') (Code generation failed for method '[System.Linq.Parallel]System.Linq.Parallel.BinaryQueryOperator`3+BinaryQueryOperatorResults+RightChildResultsRecipient`1<System.__Canon,System.__Canon,System.__Canon,System.Linq.Parallel.ConcatKey`2<System.__Canon,System.Linq.Parallel.Pair`2<System.Linq.Parallel.ConcatKey`2<System.__Canon,System.__Canon>,int32>>>.Receive<Pair`2<ConcatKey`2<__Canon,Pair`2<int32,int32>>,Pair`2<Pair`2<__Canon,int32>,Pair`2<int32,int32>>>>(PartitionedStream`2<__Canon,Pair`2<ConcatKey`2<__Canon,Pair`2<int32,int32>>,Pair`2<Pair`2<__Canon,int32>,Pair`2<int32,int32>>>>)') (Code generation failed for method '[System.Linq.Parallel]System.Linq.Parallel.BinaryQueryOperator`3+BinaryQueryOperatorResults+RightChildResultsRecipient`1<System.__Canon,System.__Canon,System.__Canon,System.Linq.Parallel.ConcatKey`2<int32,System.Linq.Parallel.Pair`2<int32,System.Linq.Parallel.Pair`2<bool,System.__Canon>>>>.Receive<ConcatKey`2<Pair`2<__Canon,Pair`2<__Canon,int32>>,ConcatKey`2<int32,ConcatKey`2<__Canon,__Canon>>>>(PartitionedStream`2<__Canon,ConcatKey`2<Pair`2<__Canon,Pair`2<__Canon,int32>>,ConcatKey`2<int32,ConcatKey`2<__Canon,__Canon>>>>)') (Code generation failed for method '[System.Linq.Parallel]System.Linq.Parallel.BinaryQueryOperator`3+BinaryQueryOperatorResults+RightChildResultsRecipient`1<System.__Canon,System.__Canon,System.__Canon,System.Linq.Parallel.Pair`2<System.Linq.Parallel.Pair`2<System.__Canon,System.Linq.Parallel.Pair`2<bool,System.__Canon>>,System.Linq.Parallel.ConcatKey`2<int32,System.Linq.Parallel.Pair`2<int32,int32>>>>.Receive<Pair`2<ConcatKey`2<int32,__Canon>,Pair`2<bool,__Canon>>>(PartitionedStream`2<__Canon,Pair`2<ConcatKey`2<int32,__Canon>,Pair`2<bool,__Canon>>>)') (Arithmetic operation resulted in an overflow.) (Code generation failed for method '[System.Linq.Parallel]System.Linq.Parallel.UnionQueryOperator`1<System.__Canon>.WrapPartitionedStreamFixedBothTypes<Pair`2<int32,Pair`2<bool,Pair`2<int32,int32>>>,Pair`2<ConcatKey`2<__Canon,int32>,Pair`2<__Canon,int32>>>(PartitionedStream`2<Pair`2<__Canon,NoKeyMemoizationRequired>,Pair`2<int32,Pair`2<bool,Pair`2<int32,int32>>>>,PartitionedStream`2<Pair`2<__Canon,NoKeyMemoizationRequired>,Pair`2<ConcatKey`2<__Canon,int32>,Pair`2<__Canon,int32>>>,IPartitionedStreamRecipient`1<__Canon>,int32,CancellationToken)')
 ---> System.OverflowException: Arithmetic operation resulted in an overflow.
   at Internal.Runtime.CompilerHelpers.ThrowHelpers.ThrowOverflowException() + 0x28
   at Internal.TypeSystem.LockFreeReaderHashtable`2.Expand(TValue[]) + 0x1d0
   at Internal.TypeSystem.LockFreeReaderHashtable`2.AddOrGetExistingInner(TValue, Boolean&) + 0x74
   at Internal.TypeSystem.LockFreeReaderHashtable`2.CreateValueAndEnsureValueIsInTable(TKey) + 0x34
   at Internal.TypeSystem.MethodDesc.InstantiateSignature(Instantiation, Instantiation) + 0x394
   at Internal.IL.InstantiatedMethodIL.GetObject(Int32, NotFoundBehavior) + 0x64
   at ILCompiler.SubstitutedILProvider.GetMethodILWithInlinedSubstitutions(MethodIL) + 0x10f4
   at ILCompiler.SubstitutedILProvider.GetMethodIL(MethodDesc) + 0x80
   at ILCompiler.Compilation.CombinedILProvider.GetMethodIL(MethodDesc) + 0x2c
   at ILCompiler.Compilation.ILCache.CreateValueFromKey(MethodDesc) + 0x48
   at Internal.TypeSystem.LockFreeReaderHashtable`2.CreateValueAndEnsureValueIsInTable(TKey) + 0x20
   at ILCompiler.Compilation.GetMethodIL(MethodDesc) + 0xd0
   at Internal.JitInterface.CorInfoImpl.CompileMethod(MethodCodeNode, MethodIL) + 0x44
   at ILCompiler.RyuJitCompilation.CompileSingleMethod(CorInfoImpl, MethodCodeNode) + 0xa0
   at ILCompiler.RyuJitCompilation.CompileSingleMethod(MethodCodeNode methodCodeNodeNeedingCode) + 0xa4
   at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`2.<ForWorker>b__1(RangeWorker& currentWorker, Int64 timeout, Boolean& replicationDelegateYieldedBeforeCompletion) + 0x2dc
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`2.<ForWorker>b__1(RangeWorker& currentWorker, Int64 timeout, Boolean& replicationDelegateYieldedBeforeCompletion) + 0x524
   at System.Threading.Tasks.TaskReplicator.Replica.Execute() + 0x80
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.TaskReplicator.Run[TState](TaskReplicator.ReplicatableUserAction`1, ParallelOptions, Boolean) + 0x2c4
   at System.Threading.Tasks.Parallel.ForWorker[TLocal,TInt](TInt, TInt, ParallelOptions, Action`1, Action`2, Func`4, Func`1, Action`1) + 0x210
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.ThrowSingleCancellationExceptionOrOtherException(ICollection, CancellationToken, Exception) + 0x30
   at System.Threading.Tasks.Parallel.ForWorker[TLocal,TInt](TInt, TInt, ParallelOptions, Action`1, Action`2, Func`4, Func`1, Action`1) + 0x430
   at ILCompiler.RyuJitCompilation.CompileMultiThreaded(List`1) + 0x1e4
   at ILCompiler.RyuJitCompilation.ComputeDependencyNodeDependencies(List`1 obj) + 0x1ac
   at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.ComputeMarkedNodes() + 0x98
   at ILCompiler.RyuJitCompilation.CompileInternal(String, ObjectDumper) + 0x30
   at ILCompiler.Compilation.ILCompiler.ICompilation.Compile(String, ObjectDumper) + 0x80
   at ILCompiler.Program.Run() + 0x25f4
   at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass236_0.<.ctor>b__0(ParseResult result) + 0x330
 ---> (Inner Exception #1) System.OverflowException: Arithmetic operation resulted in an overflow.
   at Internal.Runtime.CompilerHelpers.ThrowHelpers.ThrowOverflowException() + 0x28
   at Internal.TypeSystem.LockFreeReaderHashtable`2.Expand(TValue[]) + 0x1d0
   at Internal.TypeSystem.LockFreeReaderHashtable`2.AddOrGetExistingInner(TValue, Boolean&) + 0x74
   at Internal.TypeSystem.LockFreeReaderHashtable`2.CreateValueAndEnsureValueIsInTable(TKey) + 0x34
   at Internal.TypeSystem.MethodDesc.InstantiateSignature(Instantiation, Instantiation) + 0x394
   at Internal.IL.InstantiatedMethodIL.GetObject(Int32, NotFoundBehavior) + 0x64
   at ILCompiler.SubstitutedILProvider.GetMethodILWithInlinedSubstitutions(MethodIL) + 0x10f4
   at ILCompiler.SubstitutedILProvider.GetMethodIL(MethodDesc) + 0x80
   at ILCompiler.Compilation.CombinedILProvider.GetMethodIL(MethodDesc) + 0x2c
   at ILCompiler.Compilation.ILCache.CreateValueFromKey(MethodDesc) + 0x48
   at Internal.TypeSystem.LockFreeReaderHashtable`2.CreateValueAndEnsureValueIsInTable(TKey) + 0x20
   at ILCompiler.Compilation.GetMethodIL(MethodDesc) + 0xd0
   at Internal.JitInterface.CorInfoImpl.CompileMethod(MethodCodeNode, MethodIL) + 0x44
   at ILCompiler.RyuJitCompilation.CompileSingleMethod(CorInfoImpl, MethodCodeNode) + 0xa0
   at ILCompiler.RyuJitCompilation.CompileSingleMethod(MethodCodeNode methodCodeNodeNeedingCode) + 0xa4
   at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`2.<ForWorker>b__1(RangeWorker& currentWorker, Int64 timeout, Boolean& replicationDelegateYieldedBeforeCompletion) + 0x2dc
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`2.<ForWorker>b__1(RangeWorker& currentWorker, Int64 timeout, Boolean& replicationDelegateYieldedBeforeCompletion) + 0x524
   at System.Threading.Tasks.TaskReplicator.Replica.Execute() + 0x80<---

Regression?

Yes, this started happening in a maestro bump (dotnet/macios#20249).

So looks like somewhere in this range: 5e603d5...8510651

Known Workarounds

No response

Configuration

No response

Other information

No response

[AndyAyersMS]

Doesn't seem like a codegen issue... reclassifying.

[MichalStrehovsky]

Known generic recursion in the library: #92329.

Apparently this time the compiler is running into it in a way that it doesn't generate runtime-throwing code.

[MichalStrehovsky]

This is because the compiler was asked to root System.Linq.Parallel (--root argument in the attached repro). Simplified repro is just to add:

<ItemGroup>
  <TrimmerRootAssembly Include="System.Linq.Parallel" />
</ItemGroup>

to a hello world project file.

Not clear why this is showing up as a recent regression. Rooting System.Linq.Parallel will kill the 8.0 compiler as well.

@rolfbjarne is there something in the version bump that newly makes us pass --root to the compiler? This doesn't look like a regression that would come from here. We haven't done any changes to ILC targets for a while. The regression is wherever we started rooting the assembly.

[rolfbjarne]

OK, the actual issue I ran into on CI is different, and then I didn't notice that my attempted repro was wrong, ending up filing up this.

Filed new issue: #99611