[CI] Introduce concurrency in package testing

[ycombinator]

Currently we run pipeline tests for all packages first, followed by system tests for all packages. As the number of packages grows, as we define pipeline and system tests for more packages, and when we add a third type of test (asset loading tests), this serial ordering will slow down CI jobs.

An initial optimization might be to run each type of test for all packages concurrently. So if we have N packages and M test types, there will be M concurrent sub-jobs, one for running each type of test on the N packages.

A further optimization might be to run each type of test (pipeline, system, etc.) for each package concurrently. So if we have N packages and M test types, there will be N x M concurrent sub-jobs, one for running each type of test on each package.

[mtojek]

Be aware that changes in elastic/package-spec#31 may not be thread-safe.

[ycombinator]

Thanks @mtojek. My understanding is that the concurrency being described in this issue will result in multiple, concurrently-running processes so we shouldn't be affected by the non-thread-safe changes in elastic/package-spec#31. Still, it's good to be aware of the current limitations 👍.