ASP.NET CI Broken on Linux ARM and Linux musl-x64

[sebastienros]

Current ASP.NET CI is failing on Linux ARM and linux-musl-x64, with this stack trace:

Appeared when upgrading from 8.0.100-preview.7.23360.1 to 8.0.100-rc.1.23377.2

  Unhandled exception: System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.) (Object reference not set to an instance of an object.)
   ---> System.NullReferenceException: Object reference not set to an instance of an object.
     at System.Object.GetType()
     at System.Xml.Serialization.CodeGenerator.GetVariableType(Object var)
     at System.Xml.Serialization.XmlSerializationReaderILGen.WriteSourceEnd(String source, Type elementType, Type stackType)
     at System.Xml.Serialization.XmlSerializationReaderILGen.WriteElement(String source, String arrayName, String choiceSource, ElementAccessor element, ChoiceIdentifierAccessor choice, String checkSpecified, Boolean checkForNull, Boolean readOnly, Int32 fixupIndex, Int32 elementIndex)
     at System.Xml.Serialization.XmlSerializationReaderILGen.WriteMemberElementsIf(Member[] members, Member anyElement, String elementElseString)
     at System.Xml.Serialization.XmlSerializationReaderILGen.WriteMemberElements(Member[] members, String elementElseString, String elseString, Member anyElement, Member anyText)
     at System.Xml.Serialization.XmlSerializationReaderILGen.WriteArray(String source, String arrayName, ArrayMapping arrayMapping, Boolean readOnly, Boolean isNullable, Int32 elementIndex)
     at System.Xml.Serialization.XmlSerializationReaderILGen.WriteElement(String source, String arrayName, String choiceSource, ElementAccessor element, ChoiceIdentifierAccessor choice, String checkSpecified, Boolean checkForNull, Boolean readOnly, Int32 fixupIndex, Int32 elementIndex)
     at System.Xml.Serialization.XmlSerializationReaderILGen.WriteMemberElementsIf(Member[] members, Member anyElement, String elementElseString)
     at System.Xml.Serialization.XmlSerializationReaderILGen.WriteMemberElements(Member[] members, String elementElseString, String elseString, Member anyElement, Member anyText)
     at System.Xml.Serialization.XmlSerializationReaderILGen.WriteLiteralStructMethod(StructMapping structMapping)
     at System.Xml.Serialization.XmlSerializationILGen.GenerateReferencedMethods()
     at System.Xml.Serialization.XmlSerializationReaderILGen.GenerateEnd()
     at System.Xml.Serialization.TempAssembly.GenerateRefEmitAssembly(XmlMapping[] xmlMappings, Type[] types)
     at System.Xml.Serialization.TempAssembly..ctor(XmlMapping[] xmlMappings, Type[] types, String defaultNamespace, String location)
     at System.Xml.Serialization.XmlSerializer.GenerateTempAssembly(XmlMapping xmlMapping, Type type, String defaultNamespace, String location)
     at System.Xml.Serialization.XmlSerializer..ctor(Type type, String defaultNamespace)
     at Microsoft.DotNet.ToolPackage.ToolConfigurationDeserializer.Deserialize(String pathToXml)
     at Microsoft.DotNet.ToolPackage.ToolPackageInstance.DeserializeToolConfiguration(LockFileTargetLibrary library)
     at Microsoft.DotNet.ToolPackage.ToolPackageInstance.GetToolConfiguration()
     at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
     at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
     at System.Lazy`1.CreateValue()
     at Microsoft.DotNet.ToolPackage.ToolPackageInstance.GetCommands()
     at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
     at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
     at System.Lazy`1.CreateValue()
     at Microsoft.DotNet.Tools.Tool.Restore.ToolRestoreCommand.InstallPackages(ToolManifestPackage package, Nullable`1 configFile)
     at System.Linq.Parallel.SelectQueryOperator`2.SelectQueryOperatorResults.GetElement(Int32 index)
     at System.Linq.Parallel.QueryResults`1.get_Item(Int32 index)
     at System.Linq.Parallel.PartitionedDataSource`1.ListContiguousIndexRangeEnumerator.MoveNext(T& currentElement, Int32& currentKey)
     at System.Linq.Parallel.StopAndGoSpoolingTask`2.SpoolingWork()
     at System.Linq.Parallel.SpoolingTaskBase.Work()
     at System.Linq.Parallel.QueryTask.BaseWork(Object unused)
     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
  --- End of stack trace from previous location ---
     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
     at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
     --- End of inner exception stack trace ---
     at System.Linq.Parallel.QueryTaskGroupState.QueryEnd(Boolean userInitiatedDispose)
     at System.Linq.Parallel.SpoolingTask.SpoolStopAndGo[TInputOutput,TIgnoreKey](QueryTaskGroupState groupState, PartitionedStream`2 partitions, SynchronousChannel`1[] channels, TaskScheduler taskScheduler)
     at System.Linq.Parallel.DefaultMergeHelper`2.System.Linq.Parallel.IMergeHelper<TInputOutput>.Execute()
     at System.Linq.Parallel.MergeExecutor`1.Execute()
     at System.Linq.Parallel.MergeExecutor`1.Execute[TKey](PartitionedStream`2 partitions, Boolean ignoreOutput, ParallelMergeOptions options, TaskScheduler taskScheduler, Boolean isOrdered, CancellationState cancellationState, Int32 queryId)
     at System.Linq.Parallel.PartitionedStreamMerger`1.Receive[TKey](PartitionedStream`2 partitionedStream)
     at System.Linq.Parallel.UnaryQueryOperator`2.UnaryQueryOperatorResults.GivePartitionedStream(IPartitionedStreamRecipient`1 recipient)
     at System.Linq.Parallel.QueryOperator`1.ExecuteAndGetResultsAsArray()
     at System.Linq.ParallelEnumerable.ToArray[TSource](ParallelQuery`1 source)
     at Microsoft.DotNet.Tools.Tool.Restore.ToolRestoreCommand.Execute()
     at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
     at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, TimeSpan startupTime, ITelemetry telemetryClient)
   ---> (Inner Exception dotnet/sdk#1) System.NullReferenceException: Object reference not set to an instance of an object.
     at System.Object.GetType()
     at System.Xml.Serialization.CodeGenerator.GetVariableType(Object var)
     at System.Xml.Serialization.XmlSerializationReaderILGen.WriteSourceEnd(String source, Type elementType, Type stackType)
     at System.Xml.Serialization.XmlSerializationReaderILGen.WriteElement(String source, String arrayName, String choiceSource, ElementAccessor element, ChoiceIdentifierAccessor choice, String checkSpecified, Boolean checkForNull, Boolean readOnly, Int32 fixupIndex, Int32 elementIndex)
     at System.Xml.Serialization.XmlSerializationReaderILGen.WriteMemberElementsIf(Member[] members, Member anyElement, String elementElseString)
     at System.Xml.Serialization.XmlSerializationReaderILGen.WriteMemberElements(Member[] members, String elementElseString, String elseString, Member anyElement, Member anyText)
     at System.Xml.Serialization.XmlSerializationReaderILGen.WriteArray(String source, String arrayName, ArrayMapping arrayMapping, Boolean readOnly, Boolean isNullable, Int32 elementIndex)
     at System.Xml.Serialization.XmlSerializationReaderILGen.WriteElement(String source, String arrayName, String choiceSource, ElementAccessor element, ChoiceIdentifierAccessor choice, String checkSpecified, Boolean checkForNull, Boolean readOnly, Int32 fixupIndex, Int32 elementIndex)
     at System.Xml.Serialization.XmlSerializationReaderILGen.WriteMemberElementsIf(Member[] members, Member anyElement, String elementElseString)
     at System.Xml.Serialization.XmlSerializationReaderILGen.WriteMemberElements(Member[] members, String elementElseString, String elseString, Member anyElement, Member anyText)
     at System.Xml.Serialization.XmlSerializationReaderILGen.WriteLiteralStructMethod(StructMapping structMapping)
     at System.Xml.Serialization.XmlSerializationILGen.GenerateReferencedMethods()
     at System.Xml.Serialization.XmlSerializationReaderILGen.GenerateEnd()
     at System.Xml.Serialization.TempAssembly.GenerateRefEmitAssembly(XmlMapping[] xmlMappings, Type[] types)
     at System.Xml.Serialization.TempAssembly..ctor(XmlMapping[] xmlMappings, Type[] types, String defaultNamespace, String location)
     at System.Xml.Serialization.XmlSerializer.GenerateTempAssembly(XmlMapping xmlMapping, Type type, String defaultNamespace, String location)
     at System.Xml.Serialization.XmlSerializer..ctor(Type type, String defaultNamespace)
     at Microsoft.DotNet.ToolPackage.ToolConfigurationDeserializer.Deserialize(String pathToXml)
     at Microsoft.DotNet.ToolPackage.ToolPackageInstance.DeserializeToolConfiguration(LockFileTargetLibrary library)
     at Microsoft.DotNet.ToolPackage.ToolPackageInstance.GetToolConfiguration()
     at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
     at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
     at System.Lazy`1.CreateValue()
     at Microsoft.DotNet.ToolPackage.ToolPackageInstance.GetCommands()
     at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
     at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
     at System.Lazy`1.CreateValue()
     at Microsoft.DotNet.Tools.Tool.Restore.ToolRestoreCommand.InstallPackages(ToolManifestPackage package, Nullable`1 configFile)
     at System.Linq.Parallel.SelectQueryOperator`2.SelectQueryOperatorResults.GetElement(Int32 index)
     at System.Linq.Parallel.QueryResults`1.get_Item(Int32 index)
     at System.Linq.Parallel.PartitionedDataSource`1.ListContiguousIndexRangeEnumerator.MoveNext(T& currentElement, Int32& currentKey)
     at System.Linq.Parallel.StopAndGoSpoolingTask`2.SpoolingWork()
     at System.Linq.Parallel.SpoolingTaskBase.Work()
     at System.Linq.Parallel.QueryTask.BaseWork(Object unused)
     at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
  --- End of stack trace from previous location ---
     at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
     at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
<---
  
/home/vsts/work/1/s/.packages/microsoft.dotnet.arcade.sdk/8.0.0-beta.23371.1/tools/Tools.proj(55,5): error MSB3073: The command ""/home/vsts/work/1/s/.dotnet/dotnet" tool restore" exited with code 1.
##[error].packages/microsoft.dotnet.arcade.sdk/8.0.0-beta.23371.1/tools/Tools.proj(55,5): error MSB3073: (NETCORE_ENGINEERING_TELEMETRY=Restore) The command ""/home/vsts/work/1/s/.dotnet/dotnet" tool restore" exited with code 1.

Build FAILED.

/home/vsts/work/1/s/.packages/microsoft.dotnet.arcade.sdk/8.0.0-beta.23371.1/tools/Tools.proj(55,5): error MSB3073: The command ""/home/vsts/work/1/s/.dotnet/dotnet" tool restore" exited with code 1.

From:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=355133&view=logs&j=4a1f0fe1-d87a-57fd-c29d-71ea15e38f30&t=e34518a7-7ef2-58fc-776d-024f679b17d8

[eerhardt]

@krwq @jeffhandley - for XML since it is the place an exception is being thrown

@aortiz-msft @nkolev92 - for NuGet, since this happens during dotnet restore

[ericstj]

It seems to be happening in tool restore, where the SDK is trying to use XmlSerializer to deserialize the tool XML that it pulled out of a downloaded package. I'm not seeing any suspicious changes in XML-serialization (cc @mconnew @StephenMolloy @HongGit) or the SDK (@nagilson), nor am I seeing a reason why this code might fail in this way only on ARM. I reran the build leg and it succeeded.

[ericstj]

One thing interesting about that callstack is that it's running these installs in parallel. I wonder if there's some hidden race condition where state is shared between calls to XmlSerializer. I couldn't easily spot it in the XML serialization API - perhaps it's somehow sharing the generated assembly and that causes some sort of race. 🤷‍♂️

[mconnew]

This isn't going to be a race condition issue. I believe this is either JIT or CLR runtime bug. This is the code for CodeGenerator.GetVariableType:

runtime/src/libraries/System.Private.Xml/src/System/Xml/Serialization/CodeGenerator.cs

Lines 161 to 169 in 25fb822

	internal static Type GetVariableType(object var)
	{
	if (var is ArgBuilder)
	return ((ArgBuilder)var).ArgType;
	else if (var is LocalBuilder)
	return ((LocalBuilder)var).LocalType;
	else
	return var.GetType();
	}

        internal static Type GetVariableType(object var)
        {
            if (var is ArgBuilder)
                return ((ArgBuilder)var).ArgType;
            else if (var is LocalBuilder)
                return ((LocalBuilder)var).LocalType;
            else
                return var.GetType();
        }

If you look at the call stack, it's not throwing a NRE inside this method, the last line is in Type.GetType().

 Unhandled exception: System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.) (Object reference not set to an instance of an object.)
   ---> System.NullReferenceException: Object reference not set to an instance of an object.
     at System.Object.GetType()
     at System.Xml.Serialization.CodeGenerator.GetVariableType(Object var)

For it to have a stacktrace line for the GetType() method, the var variable must be non-null as you can't enter an instance method of a null reference. The only way to get to the line which calls var.GetType() is if var is a boxed integer. The code which returns the object which is passed to GetVariableType can only produce ArgBuilder, LocalBuilder, or the result of int32.TryParse which is then boxed by assigning to a local object variable.

[jeffschwMSFT]

@mangod9 could you take a look?

[mangod9]

hi @sebastienros, is there a repro and/or dump available to investigate this further?

[eerhardt]

@mangod9 - dotnet/aspnetcore#49763 should be repro'ing the failure.

You can see one build failure here:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=357885&view=logs&j=4a1f0fe1-d87a-57fd-c29d-71ea15e38f30&t=e34518a7-7ef2-58fc-776d-024f679b17d8&l=50

Fails on Linux ARM.

[mangod9]

Ok will take a look. Does it need to build on Linux ARM machine, or is it a cross build from x64?