[ci] Parallelize and reduce overhead of MSBuild test stage. #7850
Conversation
jpobst
changed the title
jpobst
force-pushed
the
msbuild-test-parallel
branch
from
36df0f8 to
0e3c2a3
Compare
jpobst
force-pushed
the
msbuild-test-parallel
branch
from
0e3c2a3 to
c16e1a4
Compare
jpobst
force-pushed
the
msbuild-test-parallel
branch
from
0a67b86 to
4f9e917
Compare
jpobst
force-pushed
the
msbuild-test-parallel
branch
3 times, most recently
from
01a3fee to
a9476f6
Compare
jpobst
force-pushed
the
msbuild-test-parallel
branch
from
a9476f6 to
189c376
Compare
jpobst
requested review from
dellis1972,
jonathanpeppers,
jonpryor and
pjcollins
as code owners
| jobName: win_msbuild_tests | ||
| jobDisplayName: Windows > Tests > MSBuild | ||
| agentCount: 4 | ||
| testFilter: cat != SmokeTests |
There was a problem hiding this comment.
I think we could move this block back into azure-pipelines.yaml to help cut down on the number of templates we use that aren't referenced more than once.
There was a problem hiding this comment.
My thought was that azure-pipelines.yaml is way too big and it takes too long to find the relevant stage yaml. If each stage was its own template file (stage-*.yaml) we could just go directly to the stage we want.
But that's a stylistic opinion and I'm not committed to it if others feel different. 😁
There was a problem hiding this comment.
I wish yaml had some support around "go to reference" or "go to implementation", I'm often annoyed when working with multiple layers of nested templates having to jump back and forth between multiple page tabs rather than scrolling. That's also just my opinion though and am happy to go either way on this, maybe I can find some VS code extension to make my life easier...
| - name: ExcludedNUnitCategories | ||
| value: '& cat != DotNetIgnore & cat != HybridAOT & cat != MkBundle & cat != MonoSymbolicate & cat != PackagesConfig & cat != StaticProject & cat != SystemApplication' |
There was a problem hiding this comment.
This seems duplicate of DotNetNUnitCategories, why is the syntax here different?
There was a problem hiding this comment.
dotnet test uses one syntax, NUnit uses a different syntax.
When you use dotnet test the adapter does the translation. Because dotnet-test-slicer hits NUnit.Engine directly, we need to use NUnit's test selection language.
I put a little note here https://github.com/xamarin/xamarin-android/blob/f0a24bf67b3f0e56ed7a1d24c2e3789f542f0908/build-tools/automation/yaml-templates/run-sliced-nunit-tests.yaml#L3, but it doesn't bubble up to that variable specification.
* main: [tests] `InstallAndroidDependenciesTest` uses `platform-tools` 34.0.1 (dotnet#7873) [ci] Parallelize and reduce overhead of MSBuild test stage. (dotnet#7850) [Xamarin.Android.Build.Tasks] Remove MAM targets from CoreResolve (dotnet#7868)
* main: [tests] `InstallAndroidDependenciesTest` uses `platform-tools` 34.0.1 (dotnet#7873) [ci] Parallelize and reduce overhead of MSBuild test stage. (dotnet#7850) [Xamarin.Android.Build.Tasks] Remove MAM targets from CoreResolve (dotnet#7868)
* main: [tests] `InstallAndroidDependenciesTest` uses `platform-tools` 34.0.1 (dotnet#7873) [ci] Parallelize and reduce overhead of MSBuild test stage. (dotnet#7850) [Xamarin.Android.Build.Tasks] Remove MAM targets from CoreResolve (dotnet#7868)
Context: #7850 (comment) With the re-parallelization of our test suites (#7804, #7850), combined with the fact that most PR's end up running the full test suite anyways, the separate "Smoke Tests" jobs aren't really saving us much, if any, CI time. We feel we would be better off using those additional agents to run the full test suite faster. This PR removes the separate "Smoke Tests" jobs and runs all tests in the same jobs. Additionally, as the `[Category ("SmokeTests")]` NUnit attribute is now only used by the Windows build stage to run in-tree tests, it has been cut back to the absolute bare minimum, saving ~45 minutes. Note that all the tests will continue to be run against the Windows installer, this only runs fewer tests for the Windows in-tree tests.
Bring the AzDO parallelization from #7804 and environment setup improvements from #7832 to
Xamarin.Android.Build.Tests.csprojbased test suites. This includes both the mainMSBuildstage and theSmoke Testsjobs.Increases parallelization of all jobs as many were approaching ~90 minutes.
As there is no longer a place in the
MSBuildstage to runXamarin.Android.Tools.Aidl-Teststests, it was moved to themacOS > Tests > APKs .NETstage. This suite should be fine to only run on Mac and not Windows. (Note this test assembly was also updated to .NET 7. This required moving it fromXamarin.Android-Tests.slnwhich is currently built with Mono which cannot build .NET 7+ projects. It now is built viaXamarin.Android.sln.)Ensure test count stayed the same
main4 excess tests are
DotNetPublish(True,["net8.0", null, 33]), which gets skipped anyways because the platform isnull2 excess tests are
MauiTargetFramework("net8.0",null,33), which gets skipped anyways because the platform isnullResults
Before:
After:
Note runtimes are not very stable. Sometimes you get an agent that is significantly slower than normal. 🤷♂️