Multi agent parallel testing in CI

[majocha]

Make #18517 work.

[github-actions]

✅ No release notes required

[majocha]

This kind of worked, but now Linux is the bottleneck, oh well :)

[T-Gro]

Linux and MacOs will likely alternate in being the slowest, MacOs also has the smallest pool of available machines.

I do like the numbers very much.
I will be happy to get this in on a per-partes basis ;; always focusing on the currently slowest leg(s).

[majocha]

Now it's important to verify the that number of tests add up correctly 😅

[T-Gro]

.. number of unique executed tests.

I would recommend to download testlog.xml files and have a script count unique test entries and compare two sets (prior exection, new execution from 4 jobs)

[majocha]

So I had some fun splitting the build and test phases into separate jobs.
Nominally the run times are nice, but it doesn't seem very beneficial as for now, because there is a significant wait time when starting a windows vm agent.

[majocha]

I verified that in fact all tests do have a batch number trait, and none are left out when run in batches.

The failsafe, in case this ever becomes problematic, is to remove the -testBatch x from script invocations and it will run all the tests without filtering.

As for the build / test split into separate jobs, I'm not sold on this. it's a matter of reverting the single commit.

[majocha]

For future reference:

[T-Gro]

As for the build / test split into separate jobs, I'm not sold on this. it's a matter of reverting the single commit.

Indeed, the overall time of the CI (and not duration of individual jobs) is the same, if not slower.
If we ever feel the pressure to reduce the resource utilization here, it is a step we could undertake.

=> I think we can keep each leg do it's own build.

[majocha]

Indeed, the overall time of the CI (and not duration of individual jobs) is the same, if not slower.

Yes much of this comes from waiting for the windows vms to get running. There is significant wait time between the build and test phases because of this.

=> I think we can keep each leg do it's own build.

I'm away but I'll restore this when I get back 🙂

[T-Gro]

Indeed, the overall time of the CI (and not duration of individual jobs) is the same, if not slower.

Yes much of this comes from waiting for the windows vms to get running. There is significant wait time between the build and test phases because of this.

=> I think we can keep each leg do it's own build.

I'm away but I'll restore this when I get back 🙂

Thanks @majocha . Please mark it as ready for review onace that is changed, I am eager to have this in 👍

[majocha]

This is ready, overall CI run time is now around 48 minutes. It is determined by the slowest -testVs leg, which is unfortunately not parallelized at all.

[majocha]

I had no luck trying to just split VS unit tests, which are the slowest now, into batches with this approach. Tests fail, hang and so on. Theoretically it should work with less granularity: by test class instead of test case. So, there is still room for improvement.

[T-Gro]

I had no luck trying to just split VS unit tests, which are the slowest now, into batches with this approach. Tests fail, hang and so on. Theoretically it should work with less granularity: by test class instead of test case. So, there is still room for improvement.

Thanks for trying @majocha .
This might need a more manual (cherry picked) approach to figure out a working split. The tests are interdependent and work with a lot of dependencies we cannot control.

[T-Gro]

This is a big improvement for all F# contributors, I am super happy to see this being merged!
Great job @majocha