Verticals are not publishing all packages

[NikolaMilosavljevic]

I've only checked AzureLinux_x64_Cross_x64 - it publishes packages from few repos today. It used to publish many more.

The issue seems to have started with this VMR SHA: c9986621b8c673b2db87d00d57e9bf69ed1d830f

Build with previous SHA was good.

Diff: https://github.com/dotnet/dotnet/compare/7fe01d9..c998662

This seems related to changes in vertical publishing code in dotnet/sdk#47338

@jkoritzinsky

[jkoritzinsky]

@NikolaMilosavljevic now each Unified Build vertical only publishes the packages that are unique for that vertical. For verticals other than Windows_x64, this usually means that there are very few packages published. Are there any packages you see that are missing after the final join job?

[NikolaMilosavljevic]

@NikolaMilosavljevic now each Unified Build vertical only publishes the packages that are unique for that vertical. For verticals other than Windows_x64, this usually means that there are very few packages published. Are there any packages you see that are missing after the final join job?

OK. Yes, there are some missing packages, here's the full list (versions from older build):

Microsoft.Diagnostics.NETCore.Client.0.2.616707.nupkg
Microsoft.DotNet.ScenarioTests.SdkTemplateTests.10.0.0-preview.25166.107.nupkg
Microsoft.NET.Sdk.Aspire.Manifest-8.0.100.8.2.2.nupkg
NuGet.Build.Tasks.6.14.0-preview.1.32767.nupkg
NuGet.Build.Tasks.Console.6.14.0-preview.1.32767.nupkg
NuGet.Build.Tasks.Pack.6.14.0-preview.1.32767.nupkg
NuGet.CommandLine.XPlat.6.14.0-preview.1.32767.nupkg
NuGet.Commands.6.14.0-preview.1.32767.nupkg
NuGet.Common.6.14.0-preview.1.32767.nupkg
NuGet.Configuration.6.14.0-preview.1.32767.nupkg
NuGet.Credentials.6.14.0-preview.1.32767.nupkg
NuGet.DependencyResolver.Core.6.14.0-preview.1.32767.nupkg
NuGet.Frameworks.6.14.0-preview.1.32767.nupkg
NuGet.LibraryModel.6.14.0-preview.1.32767.nupkg
NuGet.PackageManagement.6.14.0-preview.1.32767.nupkg
NuGet.Packaging.6.14.0-preview.1.32767.nupkg
NuGet.ProjectModel.6.14.0-preview.1.32767.nupkg
NuGet.Protocol.6.14.0-preview.1.32767.nupkg
NuGet.Resolver.6.14.0-preview.1.32767.nupkg
NuGet.Versioning.6.14.0-preview.1.32767.nupkg
Razor.Diagnostics.Analyzers.Test.9.0.0-preview.25166.107.nupkg
RazorDeployment.9.0.0-preview.25166.107.nupkg
VS.Redist.Common.NetCore.AppHostPack.arm64_x86.10.0.10.0.0-preview.4.25176.115.nupkg
VS.Redist.Common.NetCore.AppHostPack.x64_arm64.10.0.10.0.0-preview.4.25176.115.nupkg
VS.Redist.Common.NetCore.AppHostPack.x64_x86.10.0.10.0.0-preview.4.25176.115.nupkg
VS.Redist.Common.NetCore.AppHostPack.x86_arm64.10.0.10.0.0-preview.4.25176.115.nupkg
VS.Redist.Common.NetCore.SdkPlaceholder.arm64.10.0.10.0.100-preview.4.25176.115.nupkg
VS.Redist.Common.NetCore.SdkPlaceholder.x86.10.0.10.0.100-preview.4.25176.115.nupkg
VS.Redist.Common.NetCore.Templates.arm64.10.0.10.0.100-preview.4.25176.115.nupkg
VS.Redist.Common.NetCore.Templates.x86.10.0.10.0.100-preview.4.25176.115.nupkg
VS.Redist.Common.NetCore.Toolset.arm64.10.0.10.0.100-preview.4.25176.115.nupkg
VS.Redist.Common.NetCore.Toolset.x86.10.0.10.0.100-preview.4.25176.115.nupkg

This list was discovered by diffing the package list from a build right before your PR was merged. Some of these might make sense, but some packages are certainly produced by verticals, but missing, like Microsoft.DotNet.ScenarioTests.SdkTemplateTests.10.0.0-preview.25169.101.nupkg. This is produced by Windows vertical, published to vertical artifacts, but not to PackageArtifacts, as seen in latest full build: https://dev.azure.com/dnceng/internal/_build/results?buildId=2672789&view=artifacts&pathAsName=false&type=publishedArtifacts

Perhaps the issue is that this particular package now has ci versioning: Microsoft.DotNet.ScenarioTests.SdkTemplateTests.10.0.0-ci.nupkg

[jkoritzinsky]

The NuGet packages are being tracked by #4974

The VS.Redist ones are interesting as Arcade's publishing has some support to maintain that.

I see the Razor packages in PackageArtifacts.

The diagnostics and scenario tests ones are also interesting. Not sure what's going on there.

[NikolaMilosavljevic]

The NuGet packages are being tracked by #4974

The VS.Redist ones are interesting as Arcade's publishing has some support to maintain that.

I see the Razor packages in PackageArtifacts.

The diagnostics and scenario tests ones are also interesting. Not sure what's going on there.

Scenario-tests package is important and its absence is currently blocking Installer tests. They are failing even though the pipeline shows warnings for those jobs.

[NikolaMilosavljevic]

The NuGet packages are being tracked by #4974
The VS.Redist ones are interesting as Arcade's publishing has some support to maintain that.
I see the Razor packages in PackageArtifacts.
The diagnostics and scenario tests ones are also interesting. Not sure what's going on there.

Scenario-tests package is important and its absence is currently blocking Installer tests. They are failing even though the pipeline shows warnings for those jobs.

Interestingly, scenario-tests package is missing from Windows_x64 manifest: https://artprodcus3.artifacts.visualstudio.com/Ab55de4ed-4b5a-4215-a8e4-0a0a5f71e7d8/7ea9116e-9fac-403d-b258-b31fcf1bb293/_apis/artifact/cGlwZWxpbmVhcnRpZmFjdDovL2RuY2VuZy9wcm9qZWN0SWQvN2VhOTExNmUtOWZhYy00MDNkLWIyNTgtYjMxZmNmMWJiMjkzL2J1aWxkSWQvMjY3Mjc4OS9hcnRpZmFjdE5hbWUvV2luZG93c194NjRfQXJ0aWZhY3Rz0/content?format=file&subPath=%2Fmanifests%2FRelease%2FWindows_x64.xml

[NikolaMilosavljevic]

The NuGet packages are being tracked by #4974
The VS.Redist ones are interesting as Arcade's publishing has some support to maintain that.
I see the Razor packages in PackageArtifacts.
The diagnostics and scenario tests ones are also interesting. Not sure what's going on there.

Scenario-tests package is important and its absence is currently blocking Installer tests. They are failing even though the pipeline shows warnings for those jobs.

Interestingly, scenario-tests package is missing from Windows_x64 manifest: https://artprodcus3.artifacts.visualstudio.com/Ab55de4ed-4b5a-4215-a8e4-0a0a5f71e7d8/7ea9116e-9fac-403d-b258-b31fcf1bb293/_apis/artifact/cGlwZWxpbmVhcnRpZmFjdDovL2RuY2VuZy9wcm9qZWN0SWQvN2VhOTExNmUtOWZhYy00MDNkLWIyNTgtYjMxZmNmMWJiMjkzL2J1aWxkSWQvMjY3Mjc4OS9hcnRpZmFjdE5hbWUvV2luZG93c194NjRfQXJ0aWZhY3Rz0/content?format=file&subPath=%2Fmanifests%2FRelease%2FWindows_x64.xml

Previous good build that included ScenarioTests package - Microsoft.DotNet.ScenarioTests.SdkTemplateTests.10.0.0-preview.25167.101.nupkg:

https://dev.azure.com/dnceng/internal/_build/results?buildId=2665355&view=artifacts&pathAsName=false&type=publishedArtifacts

Vertical manifests do not include nuget packages, unless they're under assets, which is only true for various *.symbols.nupkg.

[mthalman]

Scenario tests isn't a shipping thing. Why is it important for it to have its package published?

[NikolaMilosavljevic]

Scenario tests isn't a shipping thing. Why is it important for it to have its package published?

This is about publishing packages to VMR's PackageArtifacts. We do publish all Shipping and NonShipping packages there, with some exceptions - all of them seem to be regressions due to some recent changes in publishing and join-verticals steps.

Scenario-tests are consumed by installer-tests jobs, in Validation stage. We can't build this repo at that time, as it would build all dependencies, and incur the cost of building the whole VMR again.

[NikolaMilosavljevic]

@mmitche

[jkoritzinsky]

I changed the publishing to PackageArtifacts and BlobArtifacts to only publish the files we actually want to publish externally. Is scenario-tests staging the asset after we've already made the merged manifest?