APICompat is failing when being used with desktop msbuild

[ViktorHofer]

    D:\a\_work\1\vmr\.dotnet\sdk\10.0.100-alpha.1.24555.54\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ApiCompat.ValidatePackage.targets(39,5): error MSB4018: The "Microsoft.DotNet.ApiCompat.Task.ValidatePackageTask" task failed unexpectedly. [D:\a\_work\1\vmr\src\aspnetcore\src\WebEncoders\src\Microsoft.Extensions.WebEncoders.csproj]
    D:\a\_work\1\vmr\.dotnet\sdk\10.0.100-alpha.1.24555.54\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ApiCompat.ValidatePackage.targets(39,5): error MSB4018: System.MissingMethodException: Method not found: 'Void Microsoft.CodeAnalysis.CSharp.CSharpCompilationOptions..ctor(Microsoft.CodeAnalysis.OutputKind, Boolean, System.String, System.String, System.String, System.Collections.Generic.IEnumerable`1<System.String>, Microsoft.CodeAnalysis.OptimizationLevel, Boolean, Boolean, System.String, System.String, System.Collections.Immutable.ImmutableArray`1<Byte>, System.Nullable`1<Boolean>, Microsoft.CodeAnalysis.Platform, Microsoft.CodeAnalysis.ReportDiagnostic, Int32, System.Collections.Generic.IEnumerable`1<System.Collections.Generic.KeyValuePair`2<System.String,Microsoft.CodeAnalysis.ReportDiagnostic>>, Boolean, Boolean, Microsoft.CodeAnalysis.XmlReferenceResolver, Microsoft.CodeAnalysis.SourceReferenceResolver, Microsoft.CodeAnalysis.MetadataReferenceResolver, Microsoft.CodeAnalysis.AssemblyIdentityComparer, Microsoft.CodeAnalysis.StrongNameProvider, Boolean, Microsoft.CodeAnalysis.MetadataImportOptions, Microsoft.CodeAnalysis.NullableContextOptions)'. [D:\a\_work\1\vmr\src\aspnetcore\src\WebEncoders\src\Microsoft.Extensions.WebEncoders.csproj]
    D:\a\_work\1\vmr\.dotnet\sdk\10.0.100-alpha.1.24555.54\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ApiCompat.ValidatePackage.targets(39,5): error MSB4018:    at Microsoft.DotNet.ApiSymbolExtensions.AssemblySymbolLoader..ctor(Boolean resolveAssemblyReferences, Boolean includeInternalSymbols) [D:\a\_work\1\vmr\src\aspnetcore\src\WebEncoders\src\Microsoft.Extensions.WebEncoders.csproj]
    D:\a\_work\1\vmr\.dotnet\sdk\10.0.100-alpha.1.24555.54\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ApiCompat.ValidatePackage.targets(39,5): error MSB4018:    at Microsoft.DotNet.ApiSymbolExtensions.AssemblySymbolLoaderFactory.Create(Boolean shouldResolveReferences) [D:\a\_work\1\vmr\src\aspnetcore\src\WebEncoders\src\Microsoft.Extensions.WebEncoders.csproj]
    D:\a\_work\1\vmr\.dotnet\sdk\10.0.100-alpha.1.24555.54\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ApiCompat.ValidatePackage.targets(39,5): error MSB4018:    at Microsoft.DotNet.ApiCompatibility.Runner.ApiCompatRunner.CreateAssemblySymbols(IReadOnlyList`1 metadataInformation, Boolean& resolvedExternallyProvidedAssemblyReferences) [D:\a\_work\1\vmr\src\aspnetcore\src\WebEncoders\src\Microsoft.Extensions.WebEncoders.csproj]
    D:\a\_work\1\vmr\.dotnet\sdk\10.0.100-alpha.1.24555.54\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ApiCompat.ValidatePackage.targets(39,5): error MSB4018:    at Microsoft.DotNet.ApiCompatibility.Runner.ApiCompatRunner.ExecuteWorkItems() [D:\a\_work\1\vmr\src\aspnetcore\src\WebEncoders\src\Microsoft.Extensions.WebEncoders.csproj]
    D:\a\_work\1\vmr\.dotnet\sdk\10.0.100-alpha.1.24555.54\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ApiCompat.ValidatePackage.targets(39,5): error MSB4018:    at Microsoft.DotNet.ApiCompat.ValidatePackage.Run(Func`2 logFactory, Boolean generateSuppressionFile, Boolean preserveUnnecessarySuppressions, Boolean permitUnnecessarySuppressions, String[] suppressionFiles, String suppressionOutputFile, String noWarn, Boolean respectInternals, Boolean enableRuleAttributesMustMatch, String[] excludeAttributesFiles, Boolean enableRuleCannotChangeParameterName, String packagePath, Boolean runApiCompat, Boolean enableStrictModeForCompatibleTfms, Boolean enableStrictModeForCompatibleFrameworksInPackage, Boolean enableStrictModeForBaselineValidation, String baselinePackagePath, String runtimeGraph, IReadOnlyDictionary`2 packageAssemblyReferences, IReadOnlyDictionary`2 baselinePackageAssemblyReferences, String[] baselinePackageFrameworksToIgnore) [D:\a\_work\1\vmr\src\aspnetcore\src\WebEncoders\src\Microsoft.Extensions.WebEncoders.csproj]
    D:\a\_work\1\vmr\.dotnet\sdk\10.0.100-alpha.1.24555.54\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ApiCompat.ValidatePackage.targets(39,5): error MSB4018:    at Microsoft.DotNet.ApiCompat.Task.ValidatePackageTask.ExecuteCore() [D:\a\_work\1\vmr\src\aspnetcore\src\WebEncoders\src\Microsoft.Extensions.WebEncoders.csproj]
    D:\a\_work\1\vmr\.dotnet\sdk\10.0.100-alpha.1.24555.54\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ApiCompat.ValidatePackage.targets(39,5): error MSB4018:    at Microsoft.NET.Build.Tasks.TaskBase.Execute() [D:\a\_work\1\vmr\src\aspnetcore\src\WebEncoders\src\Microsoft.Extensions.WebEncoders.csproj]
    D:\a\_work\1\vmr\.dotnet\sdk\10.0.100-alpha.1.24555.54\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ApiCompat.ValidatePackage.targets(39,5): error MSB4018:    at Microsoft.DotNet.ApiCompat.Task.ValidatePackageTask.Execute() [D:\a\_work\1\vmr\src\aspnetcore\src\WebEncoders\src\Microsoft.Extensions.WebEncoders.csproj]
    D:\a\_work\1\vmr\.dotnet\sdk\10.0.100-alpha.1.24555.54\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ApiCompat.ValidatePackage.targets(39,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Execute() [D:\a\_work\1\vmr\src\aspnetcore\src\WebEncoders\src\Microsoft.Extensions.WebEncoders.csproj]
    D:\a\_work\1\vmr\.dotnet\sdk\10.0.100-alpha.1.24555.54\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ApiCompat.ValidatePackage.targets(39,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.<ExecuteInstantiatedTask>d__26.MoveNext() [D:\a\_work\1\vmr\src\aspnetcore\src\WebEncoders\src\Microsoft.Extensions.WebEncoders.csproj]
##[error].dotnet\sdk\10.0.100-alpha.1.24555.54\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ApiCompat.ValidatePackage.targets(39,5): error MSB4018: (NETCORE_ENGINEERING_TELEMETRY=Build) The "Microsoft.DotNet.ApiCompat.Task.ValidatePackageTask" task failed unexpectedly.

[dotnet-policy-service]

@dotnet/area-infrastructure-libraries a new issue has been filed in the ApiCompat area, please triage

[ViktorHofer]

cc @ericstj

[ericstj]

I think that happens when mismatched roslyn is loaded. Did you investigate it - any more repro details @ViktorHofer?

Update - just to see if I could reproduce I tried using the .NET 9 SDK with desktop MSBuild and it worked file. I see from your log that this is using the 10.0 SDK. Probably something regressed with dependencies in 10.0.

My best guess at the root cause is that APICompat is loading a newer version of System.Collections.Immutable than Roslyn is, and MSBuild doesn't have a redirect -- given I see that method signature with System.Collections.Immutable.ImmutableArray`1<Byte>.

[dotMorten]

I just started hitting this as well for the project that references the desktop sdk (the other projects/packages are fine), and I'm using 8.0.404 SDK

[ViktorHofer]

I couldn't reproduce this locally but here's a build (with binlog) that shows the failure: #45872

[ViktorHofer]

Two workarounds:

1. If you encounter this when working in dotnet/runtime, set the two properties to false:

• https://github.com/dotnet/runtime/blob/1ce82e73932b8bc98f6395b0169c21f17f71268b/Directory.Build.props#L472
• https://github.com/dotnet/runtime/blob/1ce82e73932b8bc98f6395b0169c21f17f71268b/eng/packaging.targets#L7

2. Outside of dotnet/runtime, temporarily disable APICompat when building with desktop msbuild:
   <EnablePackageValidation Condition="'$(MSBuildRuntimeType)' != 'Core'">false</EnablePackageValidation>

I will try to find the root cause.

[ericstj]

It could be that the shared AppDomain used for tasks already has the same version of roslyn loaded and bound to a different version of System.Collections.Immutable than our task uses. In the case that @dotMorten sees this could be because targeting WindowsDesktop runs another task that triggers the load. Maybe we could examine this in the debugger if someone has a repro.

If that's the case we could fix it by loading in our own appdomain, trying to preload the same assemblies as CodeAnalysis, or moving out of proc on NetFx.

[ericstj]

I've got a reduced local repro. This is the same problem we see a lot - when a task references a newer version of a dependency than is included in MSBuild's bindingRedirects. The task will load that copy just fine, but if it tries to exchange types with any other assembly that references a lower version (which is loadable) then they'll load 2 different versions in process side-by-side and blow up like this.

The version of this problem that everyone is hitting with runtime is because the runtime is using a newer SDK with a task that references the updated System.Memory. That's newer than the one in MSBuild.

@dotMorten - I wonder what your combination of SDK and MSBuild is? A binlog will show it. If you installed a newer version of the SDK than what VS would install, and that version has an updated reference to S.C.I that would explain.

[dotMorten]

@ericstj Ugh I need to remember what I was working on December 2nd to get the repro up. That's sooooo last year :-) I'll report back here if I remember