NullReferenceException when invoking the compiler (building roslyn inside the VMR)

[ViktorHofer]

2024-12-12T15:38:31.0230270Z     /Users/runner/work/1/s/.dotnet/sdk/10.0.100-alpha.1.24555.54/Roslyn/Microsoft.VisualBasic.Core.targets(44,5): error : Process terminated. System.NullReferenceException: Object reference not set to an instance of an object. [/Users/runner/work/1/s/src/roslyn/src/Compilers/VisualBasic/Portable/Microsoft.CodeAnalysis.VisualBasic.vbproj::TargetFramework=netstandard2.0]
2024-12-12T15:38:31.0250710Z ##[error].dotnet/sdk/10.0.100-alpha.1.24555.54/Roslyn/Microsoft.VisualBasic.Core.targets(44,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) Process terminated. System.NullReferenceException: Object reference not set to an instance of an object.
2024-12-12T15:38:31.0251970Z     /Users/runner/work/1/s/.dotnet/sdk/10.0.100-alpha.1.24555.54/Roslyn/Microsoft.VisualBasic.Core.targets(44,5): error :    at Microsoft.CodeAnalysis.MetadataHelpers.GetInfoForImmediateNamespaceMembers(Boolean isGlobalNamespace, Int32 namespaceNameLength, IEnumerable`1 typesByNS, StringComparer nameComparer, IEnumerable`1& types, IEnumerable`1& namespaces) [/Users/runner/work/1/s/src/roslyn/src/Compilers/VisualBasic/Portable/Microsoft.CodeAnalysis.VisualBasic.vbproj::TargetFramework=netstandard2.0]
2024-12-12T15:38:31.0254970Z ##[error].dotnet/sdk/10.0.100-alpha.1.24555.54/Roslyn/Microsoft.VisualBasic.Core.targets(44,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build)    at Microsoft.CodeAnalysis.MetadataHelpers.GetInfoForImmediateNamespaceMembers(Boolean isGlobalNamespace, Int32 namespaceNameLength, IEnumerable`1 typesByNS, StringComparer nameComparer, IEnumerable`1& types, IEnumerable`1& namespaces)
2024-12-12T15:38:31.0255720Z     /Users/runner/work/1/s/.dotnet/sdk/10.0.100-alpha.1.24555.54/Roslyn/Microsoft.VisualBasic.Core.targets(44,5): error :    at Microsoft.CodeAnalysis.VisualBasic.Symbols.Metadata.PE.PENamespaceSymbol.LoadAllMembers(IEnumerable`1 typesByNS) [/Users/runner/work/1/s/src/roslyn/src/Compilers/VisualBasic/Portable/Microsoft.CodeAnalysis.VisualBasic.vbproj::TargetFramework=netstandard2.0]
2024-12-12T15:38:31.0258160Z ##[error].dotnet/sdk/10.0.100-alpha.1.24555.54/Roslyn/Microsoft.VisualBasic.Core.targets(44,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build)    at Microsoft.CodeAnalysis.VisualBasic.Symbols.Metadata.PE.PENamespaceSymbol.LoadAllMembers(IEnumerable`1 typesByNS)
2024-12-12T15:38:31.0258820Z     /Users/runner/work/1/s/.dotnet/sdk/10.0.100-alpha.1.24555.54/Roslyn/Microsoft.VisualBasic.Core.targets(44,5): error :    at Microsoft.CodeAnalysis.VisualBasic.Symbols.Metadata.PE.PENestedNamespaceSymbol.EnsureAllMembersLoaded() [/Users/runner/work/1/s/src/roslyn/src/Compilers/VisualBasic/Portable/Microsoft.CodeAnalysis.VisualBasic.vbproj::TargetFramework=netstandard2.0]

The callstck is way longer but then GH wouldn't let me post it. See https://dnceng.visualstudio.com/7ea9116e-9fac-403d-b258-b31fcf1bb293/_apis/build/builds/2600419/logs/2704 for more details. Happened in https://dnceng.visualstudio.com/internal/_build/results?buildId=2600419&view=logs&j=3ac9c5cd-f25e-53e7-1c9b-f058b1dbaa18&t=c4239dc2-d44f-58ef-1eff-2afe47598fb8

cc @jaredpar

[jaredpar]

@ViktorHofer going to need a bit more info to reproduce this.

[ViktorHofer]

We have the above call stack (a longer version in the console log), the build console log and the binlog. There's nothing else that I can share for this failure, unfortunately.

[jaredpar]

The callstack is not really actionable though cause this doesn't repro with a normally built compiler and isn't clear how to get to this state. Do we have

1. A crash dump of the compiler?
2. A compiler log?
3. Instructions on how to get into a state where we could debug this?

Please understand: the entirety of what we know about this issue is the small bit of text in this issue. There is no other context we have

[simonferquel]

Hey @jaredpar, we got an issue that seems pretty similar at Unity, that happens from time to time.
Dotnet SDK version is 8.0.405.
Full stack trace:

Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.CodeAnalysis.MetadataHelpers.GetInfoForImmediateNamespaceMembers(Boolean isGlobalNamespace, Int32 namespaceNameLength, IEnumerable`1 typesByNS, StringComparer nameComparer, IEnumerable`1& types, IEnumerable`1& namespaces)
   at Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamespaceSymbol.LoadAllMembers(IEnumerable`1 typesByNS)
   at Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENestedNamespaceSymbol.EnsureAllMembersLoaded()
   at Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamespaceSymbol.GetNestedNamespace(ReadOnlyMemory`1 name)
   at Microsoft.CodeAnalysis.CSharp.Symbols.NamespaceSymbol.LookupNestedNamespace(ImmutableArray`1 names)
   at Microsoft.CodeAnalysis.CSharp.Symbols.NonMissingModuleSymbol.LookupTopLevelMetadataType(MetadataTypeName& emittedName)
   at Microsoft.CodeAnalysis.CSharp.Symbols.NonMissingAssemblySymbol.LookupDeclaredTopLevelMetadataTypeInModules(MetadataTypeName& emittedName)
   at Microsoft.CodeAnalysis.CSharp.Symbols.NonMissingAssemblySymbol.LookupDeclaredTopLevelMetadataType(MetadataTypeName& emittedName)
   at Microsoft.CodeAnalysis.CSharp.Symbols.AssemblySymbol.GetTopLevelTypeByMetadataName(MetadataTypeName& metadataName, AssemblyIdentity assemblyOpt, Boolean includeReferences, Boolean isWellKnownType, ValueTuple`2& conflicts, DiagnosticBag warnings, Boolean ignoreCorLibraryDuplicatedTypes)
   at Microsoft.CodeAnalysis.CSharp.Symbols.AssemblySymbol.GetTypeByMetadataName(String metadataName, Boolean includeReferences, Boolean isWellKnownType, ValueTuple`2& conflicts, Boolean useCLSCompliantNameArityEncoding, DiagnosticBag warnings, Boolean ignoreCorLibraryDuplicatedTypes)
   at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.GetWellKnownType(WellKnownType type)
   at Microsoft.CodeAnalysis.CSharp.Symbols.CSharpAttributeData.IsSecurityAttribute(CSharpCompilation compilation)
   at Microsoft.CodeAnalysis.CSharp.Symbol.ValidateAttributeUsage(CSharpAttributeData attribute, AttributeSyntax node, CSharpCompilation compilation, AttributeLocation symbolPart, BindingDiagnosticBag diagnostics, HashSet`1 uniqueAttributeTypes)
   at Microsoft.CodeAnalysis.CSharp.Symbol.ValidateAttributeUsageAndDecodeWellKnownAttributes(ImmutableArray`1 binders, ImmutableArray`1 attributeSyntaxList, ImmutableArray`1 boundAttributes, BindingDiagnosticBag diagnostics, AttributeLocation symbolPart)
   at Microsoft.CodeAnalysis.CSharp.Symbol.LoadAndValidateAttributes(OneOrMany`1 attributesSyntaxLists, CustomAttributesBag`1& lazyCustomAttributesBag, AttributeLocation symbolPart, Boolean earlyDecodingOnly, Binder binderOpt, Func`2 attributeMatchesOpt, Action`1 beforeAttributePartBound, Action`1 afterAttributePartBound)
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceAssemblySymbol.EnsureAttributesAreBound()
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceAssemblySymbol.ForceComplete(SourceLocation locationOpt, Predicate`1 filter, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.GetSourceDeclarationDiagnostics(SyntaxTree syntaxTree, Nullable`1 filterSpanWithinTree, Func`4 locationFilterOpt, Predicate`1 symbolFilter, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.GetDiagnosticsWithoutSeverityFiltering(CompilationStage stage, Boolean includeEarlierStages, BindingDiagnosticBag builder, Predicate`1 symbolFilter, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.GetDiagnostics(CompilationStage stage, Boolean includeEarlierStages, DiagnosticBag diagnostics, Predicate`1 symbolFilter, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.CommonCompiler.CompileAndEmit(TouchedFileLogger touchedFilesLogger, Compilation& compilation, ImmutableArray`1 analyzers, ImmutableArray`1 generators, ImmutableArray`1 additionalTextFiles, AnalyzerConfigSet analyzerConfigSet, ImmutableArray`1 sourceFileAnalyzerConfigOptions, ImmutableArray`1 embeddedTexts, DiagnosticBag diagnostics, ErrorLogger errorLogger, CancellationToken cancellationToken, CancellationTokenSource& analyzerCts, AnalyzerDriver& analyzerDriver, Nullable`1& generatorTimingInfo)
   at Microsoft.CodeAnalysis.CommonCompiler.RunCore(TextWriter consoleOutput, ErrorLogger errorLogger, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.CommonCompiler.Run(TextWriter consoleOutput, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc.<>c__DisplayClass1_0.<Run>b__0(TextWriter tw)
   at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc.Run(String[] args, BuildPaths buildPaths, TextWriter textWriter, IAnalyzerAssemblyLoader analyzerLoader)
   at Microsoft.CodeAnalysis.CommandLine.BuildClient.RunCompilation(IEnumerable`1 originalArguments, BuildPaths buildPaths, TextWriter textWriter, String pipeName)
   at Microsoft.CodeAnalysis.CommandLine.BuildClient.Run(IEnumerable`1 arguments, RequestLanguage language, CompileFunc compileFunc, CompileOnServerFunc compileOnServerFunc, ICompilerServerLogger logger)
   at Microsoft.CodeAnalysis.CSharp.CommandLine.Program.MainCore(String[] args)
   at Microsoft.CodeAnalysis.CSharp.CommandLine.Program.Main(String[] args)

Command line (redacted a bit):

dotnet exec "DotNetSdk/sdk/8.0.405/Roslyn/bincore/csc.dll"
/nostdlib /noconfig /shared
-target:library
-out:"Library/Bee/artifacts/200b0aEChk.dag/Unity.TestFramework.Build.dll"
-refout:"Library/Bee/artifacts/200b0aEChk.dag/Unity.TestFramework.Build.ref.dll"
<lots of defines>
<lots of references to Unity API>
<a few custom analyzer - but they don't appear in the stack trace>
"Library/PackageCache/com.unity.test-framework.build/SplitBuildAndRun.cs" (single file)
-langversion:9.0
/deterministic
/optimize+
/debug:portable
/nologo
/RuntimeMetadataVersion:v4.0.30319
/nowarn:0169
/nowarn:0649
/nowarn:0282
/nowarn:1701
/nowarn:1702
/utf8output
/preferreduilang:en-US
-warn:0

It seems to be related to validating assembly level atributes. The file in question has some custom assembly attributes:

[assembly: Preserve]

[assembly: TestPlayerBuildModifier(typeof(SplitBuildAndRun))]
[assembly: PostBuildCleanup(typeof(SplitBuildAndRun))]

With these implementations (in reference assemblies):

    [RequiredByNativeCode]
    [System.AttributeUsage(AttributeTargets.Method | AttributeTargets.Class | AttributeTargets.Field | AttributeTargets.Property
        | AttributeTargets.Constructor | AttributeTargets.Interface | AttributeTargets.Delegate | AttributeTargets.Event | AttributeTargets.Struct | AttributeTargets.Assembly | AttributeTargets.Enum, Inherited = false)]
    [System.Diagnostics.CodeAnalysis.SuppressMessageAttribute("CodeReloadSafety", "UAL0001:Unsealed Public Class", Justification = "Unsealed on purpose")]
    public class PreserveAttribute : System.Attribute
    {
    }
---
    [AttributeUsage(AttributeTargets.Assembly)]
    public class TestPlayerBuildModifierAttribute : Attribute
    {
        private Type m_Type;

        /// <summary>
        /// Initializes and returns an instance of TestPlayerBuildModifierAttribute or throws an <see cref="ArgumentException"/>.
        /// </summary>
        /// <param name="type">A target type that implements ITestPlayerBuildModifier.</param>
        /// <exception cref="ArgumentException">Throws a <see cref="ArgumentException"/> if the type provided does not implemented the `ITestPlayerBuildModifier` interface. </exception>
        public TestPlayerBuildModifierAttribute(Type type)
        {
            var interfaceType = typeof(ITestPlayerBuildModifier);
            if (!interfaceType.IsAssignableFrom(type))
            {
                throw new ArgumentException(string.Format("Type provided to {0} does not implement {1}", GetType().Name, interfaceType.Name));
            }
            m_Type = type;
        }

        internal ITestPlayerBuildModifier ConstructModifier()
        {
            return Activator.CreateInstance(m_Type) as ITestPlayerBuildModifier;
        }
    }
---
    [AttributeUsage(AttributeTargets.Assembly | AttributeTargets.Class | AttributeTargets.Method)]
    public class PostBuildCleanupAttribute : Attribute
    {
        /// <summary>
        /// Initializes and returns an instance of PostBuildCleanupAttribute by type.
        /// </summary>
        /// <param name="targetClass">The type of the target class.</param>
        public PostBuildCleanupAttribute(Type targetClass)
        {
            TargetClass = targetClass;
        }

        /// <summary>
        /// Initializes and returns an instance of PostBuildCleanupAttribute by class name.
        /// </summary>
        /// <param name="targetClassName">The name of the target class.</param>
        public PostBuildCleanupAttribute(string targetClassName)
        {
            TargetClass = AttributeHelper.GetTargetClassFromName(targetClassName, typeof(IPostBuildCleanup));
        }

        internal Type TargetClass { get; private set; }
    }

From our CI analytics, it seems to happen quite rarely, on OSX-arm64 and OSX-x64 machines only.

I hope it helps!

[simonferquel]

Also, it seems that once the failure happen, the csc build server seem to fail on all subsequent builds with the same stack trace.

[simonferquel]

Last minute details: we see failures with machines running osx 13 on arm64 and x64, as well as osx 14 on arm64. I am not sure if we have other OSX versions in our CI, but if it can help I can dig into that.

[rolfbjarne]

Looks like a duplicate: dotnet/roslyn-analyzers#7612