6.0 dotnet/runtime: Microsoft.DotNet.Compatibility.ValidatePackage fails, "System.MissingMethodException: Method not found: '...'"

[dagood]

I hit this in dotnet/runtime in the online tarball build on 2021-10-11. The conditions aren't clear, but so far it seems like flakiness:

/home/dagood/sb/6.0-tb-msbuild/src/runtime.b1aedd5a54d0c54ec12afe98a41979fb053998a5/artifacts/source-build/self/package-cache/microsoft.dotnet.compatibility/1.0.0-rc.2.21419.17/build/Microsoft.NET.Compatibility.Common.targets(26,5):
  error MSB4018: The "Microsoft.DotNet.Compatibility.ValidatePackage" task failed unexpectedly.
System.MissingMethodException: Method not found: 'Int32 Microsoft.CodeAnalysis.ISymbol.get_MetadataToken()'.
   at Microsoft.CodeAnalysis.CSharp.Symbols.AssemblySymbol.CreateISymbol()
   at Microsoft.CodeAnalysis.CSharp.Symbol.get_ISymbol()
   at Microsoft.CodeAnalysis.CSharp.Symbols.SymbolExtensions.GetPublicSymbol[TISymbol](Symbol symbol)
   at Microsoft.CodeAnalysis.CSharp.Symbols.SymbolExtensions.GetPublicSymbol(Symbol symbol)
   at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.CommonGetAssemblyOrModuleSymbol(MetadataReference reference)
   at Microsoft.CodeAnalysis.Compilation.GetAssemblyOrModuleSymbol(MetadataReference reference)
   at Microsoft.DotNet.ApiCompatibility.AssemblySymbolLoader.LoadAssembly(String name, Stream stream) in /_/src/Compatibility/Microsoft.DotNet.ApiCompatibility/AssemblySymbolLoader.cs:line 166
   at Microsoft.DotNet.PackageValidation.ApiCompatRunner.RunApiCompat() in /_/src/Compatibility/Microsoft.DotNet.PackageValidation/ApiCompatRunner.cs:line 47
   at Microsoft.DotNet.PackageValidation.CompatibleFrameworkInPackageValidator.Validate(Package package) in /_/src/Compatibility/Microsoft.DotNet.PackageValidation/CompatibleFrameworkInPackageValidator.cs:line 62
   at Microsoft.DotNet.Compatibility.ValidatePackage.ExecuteCore() in /_/src/Compatibility/Microsoft.DotNet.Compatibility/ValidatePackage.cs:line 65
   at Microsoft.NET.Build.Tasks.TaskBase.Execute() in /_/src/Tasks/Common/TaskBase.cs:line 38
   at Microsoft.DotNet.Compatibility.ValidatePackage.Execute() in /_/src/Compatibility/Microsoft.DotNet.Compatibility/ValidatePackage.cs:line 45
   at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
   at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/home/dagood/sb/6.0-tb-msbuild/src/runtime.b1aedd5a54d0c54ec12afe98a41979fb053998a5/artifacts/source-build/self/src/src/libraries/Microsoft.Extensions.Caching.Abstractions/src/Microsoft.Extensions.Caching.Abstractions.csproj]

After I hit the error, I reran ./build.sh --online in the tarball, and it passed on the second attempt.

Here's the task's subtree in the binlog, with a diff vs. the same tree in a "good" build: https://gist.github.com/dagood/b2bfb711265cd2b7138d740f2060877c

The only difference except the error itself is that the "bad" version has this line:
NoWarn = ;1701;1702;1705;1591,CS8969;NU5104;NU5104;NU5105
And the "good" version has this line... just repeating NU5104 two more times:
NoWarn = ;1701;1702;1705;1591,CS8969;NU5104;NU5104;NU5104;NU5104;NU5105

This suggests to me there must be some subtle MSBuild evaluation difference. There are some differences in the binlogs--search for $project Microsoft.Extensions.Caching.Abstractions.csproj Build netstandard2.0 net461, pick the second one (the src/ project, not ref/), and go to the evaluation node (click the id:1264 / id:1255 link on the project node), copy the subtree, and diff:
https://gist.github.com/dagood/e4abb7b2bd55ce2b0519e3978c9909e3
(Included lots of diff context lines.)

I'm not sure what the significance is, though. (GitHub isn't letting me upload files right now, so I'll try to attach the binlogs later.)

My build was based on dotnet/installer@18d694a877.

---

@eerhardt also hit this (on 2021-10-12):

I didn't change what SHA I was on for any repos. I just made some .csproj changes in the runtime, ./build.sh --online, got the above error, ./build.sh --online again and was successful
for installer, I'm on src/installer.e0343e851a6dd66bbae5abc2509a446c02847e5d

---

I searched for the task name in dotnet/source-build and I see that @omajid hit this before, here: #2477 (comment) (2021-09-26). I don't see any diagnosis in the thread though--I guess it wasn't related to the main topic in the end:

I tried replacing runtime.linux-x64.microsoft.netcore.ilasm and runtime.linux-x64.microsoft.netcore.ildasm with versions downloaded from https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet6/nuget/v3/index.json but it leads to errors in the build:

dotnet-28be3e9a006d90d8c6e87d4353b77882829df718-x64-bootstrap/src/runtime.826f81a11ad17f415668fe1cb934bdaf00d36ea2/artifacts/source-build/self/package-cache/microsoft.dotnet.compatibility/1.0.0-rc.2.21419.17/build/Microsoft.NET.Compatibility.Common.targets(26,5): error MSB4018: The "Microsoft.DotNet.Compatibility.ValidatePackage" task failed unexpectedly. [dotnet-28be3e9a006d90d8c6e87d4353b77882829df718-x64-bootstrap/src/runtime.826f81a11ad17f415668fe1cb934bdaf00d36ea2/artifacts/source-build/self/src/src/libraries/System.IO.Hashing/src/System.IO.Hashing.csproj]                                                                             
dotnet-28be3e9a006d90d8c6e87d4353b77882829df718-x64-bootstrap/src/runtime.826f81a11ad17f415668fe1cb934bdaf00d36ea2/artifacts/source-build/self/package-cache/microsoft.dotnet.compatibility/1.0.0-rc.2.21419.17/build/Microsoft.NET.Compatibility.Common.targets(26,5): error MSB4018: System.MissingMethodException: Method not found: 'Int32 Microsoft.CodeAnalysis.ISymbol.get_MetadataToken()'. [dotnet-28be3e9a006d90d8c6e87d4353b77882829df718-x64-bootstrap/src/runtime.826f81a11ad17f415668fe1cb934bdaf00d36ea2/artifacts/source-build/self/src/src/libraries/System.IO.Hashing/src/System.IO.Hashing.csproj]  

Is this a known issue? Any idea how to work around this?

---

Based on the random hits and retry seeming to always work when attempted, it seems like flakiness. The source of flakiness isn't clear though... it doesn't make sense for this build task to be nondeterministic, and the inputs appear effectively identical in the binlog.

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[dagood]

Binlogs are here for now until GitHub lets me upload them: https://teams.microsoft.com/l/message/19:e9daed71304b44cda82a845d3645b1d8@thread.skype/1634061434097?tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47&groupId=dc6359c5-e96a-44ce-9d86-0af7fab1c15e&parentMessageId=1634061434097&teamName=.NET%20Acquisition%20and%20Deployment&channelName=Source-build&createdTime=1634061434097

[dagood]

I hit this (or very similar) again, also went away with retry:

The "Microsoft.DotNet.Compatibility.ValidatePackage" task failed unexpectedly.
System.TypeLoadException: Method 'CompareSourceLocations' in type 'Microsoft.CodeAnalysis.CSharp.CSharpCompilation' from assembly 'Microsoft.CodeAnalysis.CSharp, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' does not have an implementation.
   at Microsoft.DotNet.ApiCompatibility.AssemblySymbolLoader..ctor(Boolean resolveAssemblyReferences)
   at Microsoft.DotNet.PackageValidation.ApiCompatRunner.GetAssemblySymbolFromStream(Stream assemblyStream, MetadataInformation assemblyInformation, Boolean& resolvedReferences) in /tarball/src/sdk.4d19f430e05e0c0771b6abf86be9bf137f2cc242/artifacts/source-build/self/src/src/Compatibility/Microsoft.DotNet.PackageValidation/ApiCompatRunner.cs:line 113
   at Microsoft.DotNet.PackageValidation.ApiCompatRunner.RunApiCompat() in /tarball/src/sdk.4d19f430e05e0c0771b6abf86be9bf137f2cc242/artifacts/source-build/self/src/src/Compatibility/Microsoft.DotNet.PackageValidation/ApiCompatRunner.cs:line 50
   at Microsoft.DotNet.PackageValidation.CompatibleFrameworkInPackageValidator.Validate(Package package) in /tarball/src/sdk.4d19f430e05e0c0771b6abf86be9bf137f2cc242/artifacts/source-build/self/src/src/Compatibility/Microsoft.DotNet.PackageValidation/CompatibleFrameworkInPackageValidator.cs:line 62
   at Microsoft.DotNet.Compatibility.ValidatePackage.ExecuteCore() in /tarball/src/sdk.4d19f430e05e0c0771b6abf86be9bf137f2cc242/artifacts/source-build/self/src/src/Compatibility/Microsoft.DotNet.Compatibility/ValidatePackage.cs:line 92
   at Microsoft.NET.Build.Tasks.TaskBase.Execute() in /tarball/src/sdk.4d19f430e05e0c0771b6abf86be9bf137f2cc242/artifacts/source-build/self/src/src/Tasks/Common/TaskBase.cs:line 38
   at Microsoft.DotNet.Compatibility.ValidatePackage.Execute() in /tarball/src/sdk.4d19f430e05e0c0771b6abf86be9bf137f2cc242/artifacts/source-build/self/src/src/Compatibility/Microsoft.DotNet.Compatibility/ValidatePackage.cs:line 49
   at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
   at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask)

[omajid]

FWIW, I am running into this on pretty much every build on my end. @tmds suggested adding a /p:EnablePackageValidation=false in src/runtime.*/eng/SourceBuild.props and that's been my workaround since.

[dagood]

Hmm. A few more bits of info, then: both builds were on a "real" Fedora 33 machine with an i7-6700 and SSD. (I've gotten in trouble before for having a faster-than-CI build machine, so just writing that down in case it's somehow that kind of race.)

[tmds]

I think it makes sense to skip package validation on source-build.

Afaik package validation happens against a previous version of the package, so disabling it may even get rid of some prebuilts.

@dagood @dseefeld wdyt?

[dagood]

Sounds good to me. I'll start this off with an issue to discuss the failure itself (which probably indicates some problem!) and a PR to "officially" disable it during source-build.

[dagood]

Ah, found dotnet/runtime#59908. This is apparently a known issue that has been fixed in main:

• Pin compiler version to workaround package validation loading missmatching roslyn versions runtime#60026
• Update compatibility package and unpin compiler version runtime#60847.

@safern @ViktorHofer is there a plan to get this into release/6.0 (or has it been)? We've been hitting this frequently in the 6.0 dev process. As an alternative, it seems like we can remove ValidatePackage entirely from source-build. (Depending on the Roslyn versions involved, maybe it's impossible to properly fix in 6.0?)

[ViktorHofer]

I will let @safern speak to that as getting this into 6.0 would require both the ework in dotnet/sdk and the changes that you listed above to be ported.

[safern]

@dagood I think we can port the fix on the SDK to release/6.0. This is the fix: dotnet/sdk#22277

@marcpopMSFT @mmitche you think this is something we can do for 6.0 or should we have to wait for the first servicing release?

@dagood maybe you can apply that patch on source-build in the meantime?

[dagood]

maybe you can apply that patch on source-build in the meantime?

From a source-build perspective, it seems safer to me to disable the validation in dotnet/runtime rather than port a fix from main that changes a DLL that ships in the SDK.

(Given the very limited amount of source-build tests, I think it's safest to wait for the port to be in the official dotnet/sdk servicing branch, so we are making sure the Microsoft-built SDK doesn't hit snags with the port during its more extensive set of tests.)

Or does the validation task do something important in the dotnet/runtime build beyond being a guardrail for devs?

[safern]

Or does the validation task do something important in the dotnet/runtime build beyond being a guardrail for devs?

It is a guardrail for devs, so I think disabling on source-build shouldn't affect anything, specially since it will be disabled on the 6.0 source-build so I don't expect the package validation task to catch any issues in servicing due to packages not changing nor new apis added, etc. Plus we are covered with the dotnet/runtime build.

You can just set this property to false on source-build: https://github.com/dotnet/runtime/blob/release/6.0/eng/packaging.targets#L4