Enable VMR tests in more verticals

[ViktorHofer]

Fixes dotnet/source-build#4364

• The dev versions issue is now solved since 8af4157
• The short stack and PGO builds can run tests as well (except scenario-tests).

[ViktorHofer]

/azp run sdk-unified-build-full

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[ViktorHofer]

/azp run sdk-unified-build-full

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[akoeplinger]

The short stack builds can run tests as well (except scenario-tests).

I'm not sure that makes sense. The only tests that currently run there are Microsoft.DotNet.Tests.csproj and those run on the host SDK from what I can see and don't test anything from the verticals.

[ViktorHofer]

Correct. The goal is to enable running tests on any vertical. The short stack ones obviously can't run tests that require the target SDK, but that's fine. The unit test project should run on all the verticals. We shouldn't treat some verticals more important than others in regards to unit testing the host environment.

[akoeplinger]

The unit test project should run on all the verticals. We shouldn't treat some verticals more important than others in regards to unit testing the host environment.

I don't understand, the unit test project doesn't touch any of the vertical outputs so it just wastes CI resources to run it across dozens of jobs?

[ViktorHofer]

Those tests are validating the correctness of the task assemblies that we use in the VMR orchestrator to build the product. While the tasks themselves aren't shipping output, they are component that should get tested.

I'm not sure what you mean by waste. Executing the tests only take between 5-15 seconds:

[akoeplinger]

Those tests are validating the correctness of the task assemblies that we use in the VMR orchestrator to build the product. While the tasks themselves aren't shipping output, they are component that should get tested.

Yes and those already get tested in the main Linux/OSX/Windows x64 job which is the same host as the shortstack so it is exactly the same.

And while it doesn't add a lot of time to CI it still clutters the AzDO test view and potentially increases flakiness. I don't see how this adds any value.

[ViktorHofer]

The goal here is consistency and the ability to add tests more easily in the future as many of the legs are completely untested today. Let's talk about this in today's sync to hear from others.

[ViktorHofer]

We chatted about this in our sync and decided to go ahead with this change as it will lower the barrier for future testing initiatives in these verticals.