VMR Stage 2 leg failing in arcade with dependency downgrades

[ellahathaway]

The following errors are appearing when building arcade for CentOSStream8_Online_CurrentSourceBuiltSdk_x64 leg:

    /vmr/src/arcade/artifacts/sb/src/src/Microsoft.DotNet.CodeAnalysis/Microsoft.DotNet.CodeAnalysis.csproj : error NU1109: Detected package downgrade: System.Collections.Immutable from 9.0.0-alpha.1.24061.26 to centrally defined 7.0.0. Update the centrally managed package version to a higher version.  [/vmr/src/arcade/artifacts/sb/src/Arcade.sln]
    /vmr/src/arcade/artifacts/sb/src/src/Microsoft.DotNet.CodeAnalysis/Microsoft.DotNet.CodeAnalysis.csproj : error NU1109:  Microsoft.DotNet.CodeAnalysis -> Microsoft.CodeAnalysis.CSharp 4.10.0-1.24061.4 -> Microsoft.CodeAnalysis.Common 4.10.0-1.24061.4 -> System.Collections.Immutable (>= 9.0.0-alpha.1.24061.26)  [/vmr/src/arcade/artifacts/sb/src/Arcade.sln]
    /vmr/src/arcade/artifacts/sb/src/src/Microsoft.DotNet.CodeAnalysis/Microsoft.DotNet.CodeAnalysis.csproj : error NU1109:  Microsoft.DotNet.CodeAnalysis -> System.Collections.Immutable (>= 7.0.0) [/vmr/src/arcade/artifacts/sb/src/Arcade.sln]
##[error]/vmr/src/arcade/artifacts/sb/src/src/Microsoft.DotNet.CodeAnalysis/Microsoft.DotNet.CodeAnalysis.csproj(0,0): error NU1109: (NETCORE_ENGINEERING_TELEMETRY=Restore) Detected package downgrade: System.Collections.Immutable from 9.0.0-alpha.1.24061.26 to centrally defined 7.0.0. Update the centrally managed package version to a higher version. 
 Microsoft.DotNet.CodeAnalysis -> Microsoft.CodeAnalysis.CSharp 4.10.0-1.24061.4 -> Microsoft.CodeAnalysis.Common 4.10.0-1.24061.4 -> System.Collections.Immutable (>= 9.0.0-alpha.1.24061.26) 
 Microsoft.DotNet.CodeAnalysis -> System.Collections.Immutable (>= 7.0.0)
    /vmr/src/arcade/artifacts/sb/src/src/Microsoft.DotNet.CodeAnalysis/Microsoft.DotNet.CodeAnalysis.csproj : error NU1109: Detected package downgrade: System.Reflection.Metadata from 9.0.0-alpha.1.24061.26 to centrally defined 7.0.0. Update the centrally managed package version to a higher version.  [/vmr/src/arcade/artifacts/sb/src/Arcade.sln]
    /vmr/src/arcade/artifacts/sb/src/src/Microsoft.DotNet.CodeAnalysis/Microsoft.DotNet.CodeAnalysis.csproj : error NU1109:  Microsoft.DotNet.CodeAnalysis -> Microsoft.CodeAnalysis.CSharp 4.10.0-1.24061.4 -> Microsoft.CodeAnalysis.Common 4.10.0-1.24061.4 -> System.Reflection.Metadata (>= 9.0.0-alpha.1.24061.26)  [/vmr/src/arcade/artifacts/sb/src/Arcade.sln]
    /vmr/src/arcade/artifacts/sb/src/src/Microsoft.DotNet.CodeAnalysis/Microsoft.DotNet.CodeAnalysis.csproj : error NU1109:  Microsoft.DotNet.CodeAnalysis -> System.Reflection.Metadata (>= 7.0.0) [/vmr/src/arcade/artifacts/sb/src/Arcade.sln]
##[error]/vmr/src/arcade/artifacts/sb/src/src/Microsoft.DotNet.CodeAnalysis/Microsoft.DotNet.CodeAnalysis.csproj(0,0): error NU1109: (NETCORE_ENGINEERING_TELEMETRY=Restore) Detected package downgrade: System.Reflection.Metadata from 9.0.0-alpha.1.24061.26 to centrally defined 7.0.0. Update the centrally managed package version to a higher version. 
 Microsoft.DotNet.CodeAnalysis -> Microsoft.CodeAnalysis.CSharp 4.10.0-1.24061.4 -> Microsoft.CodeAnalysis.Common 4.10.0-1.24061.4 -> System.Reflection.Metadata (>= 9.0.0-alpha.1.24061.26) 
 Microsoft.DotNet.CodeAnalysis -> System.Reflection.Metadata (>= 7.0.0)
    /vmr/src/arcade/artifacts/sb/src/src/Microsoft.DotNet.GenFacades/Microsoft.DotNet.GenFacades.csproj : error NU1109: Detected package downgrade: System.Collections.Immutable from 9.0.0-alpha.1.24061.26 to centrally defined 7.0.0. Update the centrally managed package version to a higher version.  [/vmr/src/arcade/artifacts/sb/src/Arcade.sln]
    /vmr/src/arcade/artifacts/sb/src/src/Microsoft.DotNet.GenFacades/Microsoft.DotNet.GenFacades.csproj : error NU1109:  Microsoft.DotNet.GenFacades -> Microsoft.CodeAnalysis.CSharp 4.10.0-1.24061.4 -> Microsoft.CodeAnalysis.Common 4.10.0-1.24061.4 -> System.Collections.Immutable (>= 9.0.0-alpha.1.24061.26)  [/vmr/src/arcade/artifacts/sb/src/Arcade.sln]
    /vmr/src/arcade/artifacts/sb/src/src/Microsoft.DotNet.GenFacades/Microsoft.DotNet.GenFacades.csproj : error NU1109:  Microsoft.DotNet.GenFacades -> System.Collections.Immutable (>= 7.0.0) [/vmr/src/arcade/artifacts/sb/src/Arcade.sln]
##[error]/vmr/src/arcade/artifacts/sb/src/src/Microsoft.DotNet.GenFacades/Microsoft.DotNet.GenFacades.csproj(0,0): error NU1109: (NETCORE_ENGINEERING_TELEMETRY=Restore) Detected package downgrade: System.Collections.Immutable from 9.0.0-alpha.1.24061.26 to centrally defined 7.0.0. Update the centrally managed package version to a higher version. 
 Microsoft.DotNet.GenFacades -> Microsoft.CodeAnalysis.CSharp 4.10.0-1.24061.4 -> Microsoft.CodeAnalysis.Common 4.10.0-1.24061.4 -> System.Collections.Immutable (>= 9.0.0-alpha.1.24061.26) 
 Microsoft.DotNet.GenFacades -> System.Collections.Immutable (>= 7.0.0)
    /vmr/src/arcade/artifacts/sb/src/src/Microsoft.DotNet.GenFacades/Microsoft.DotNet.GenFacades.csproj : error NU1109: Detected package downgrade: System.Reflection.Metadata from 9.0.0-alpha.1.24061.26 to centrally defined 7.0.0. Update the centrally managed package version to a higher version.  [/vmr/src/arcade/artifacts/sb/src/Arcade.sln]
    /vmr/src/arcade/artifacts/sb/src/src/Microsoft.DotNet.GenFacades/Microsoft.DotNet.GenFacades.csproj : error NU1109:  Microsoft.DotNet.GenFacades -> Microsoft.CodeAnalysis.CSharp 4.10.0-1.24061.4 -> Microsoft.CodeAnalysis.Common 4.10.0-1.24061.4 -> System.Reflection.Metadata (>= 9.0.0-alpha.1.24061.26)  [/vmr/src/arcade/artifacts/sb/src/Arcade.sln]
    /vmr/src/arcade/artifacts/sb/src/src/Microsoft.DotNet.GenFacades/Microsoft.DotNet.GenFacades.csproj : error NU1109:  Microsoft.DotNet.GenFacades -> System.Reflection.Metadata (>= 7.0.0) [/vmr/src/arcade/artifacts/sb/src/Arcade.sln]
##[error]/vmr/src/arcade/artifacts/sb/src/src/Microsoft.DotNet.GenFacades/Microsoft.DotNet.GenFacades.csproj(0,0): error NU1109: (NETCORE_ENGINEERING_TELEMETRY=Restore) Detected package downgrade: System.Reflection.Metadata from 9.0.0-alpha.1.24061.26 to centrally defined 7.0.0. Update the centrally managed package version to a higher version. 
 Microsoft.DotNet.GenFacades -> Microsoft.CodeAnalysis.CSharp 4.10.0-1.24061.4 -> Microsoft.CodeAnalysis.Common 4.10.0-1.24061.4 -> System.Reflection.Metadata (>= 9.0.0-alpha.1.24061.26) 
 Microsoft.DotNet.GenFacades -> System.Reflection.Metadata (>= 7.0.0)

Build (internal link)

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ellahathaway]

Confirmed that Microsoft.CodeAnalysis.Common v4.9.0-3.24053.1 pulls 7.0 versions of System.Collections.Immutable and System.Reflection.Metadata as expected.

Looks like dotnet/roslyn@4ce2da8 added System.Collections.Immutable and System.Reflection.Metadata to Version.Details.xml, allowing the live versions of these assemblies to flow in for Microsoft.CodeAnalysis.Common. This was done in dotnet/roslyn#71444. This is the cause of the failures that we're seeing.

cc @mthalman @MichaelSimons - Do you know whether the correct approach is to flow in the live versions for these two dependencies in Arcade (currently pinned at 7.0 in Arcade) or to remove the entries from Roslyn's Version.Details?

[mthalman]

I think you'd need to define dependencies for System.Collections.Immutable and System.Reflection.Metadata in Arcade's Version.Details.xml with a comment that these are transitive dependencies which need to be defined for source build. That should allow us to flow the N-1 version as the dependency Arcade will use instead of 7.0.0.