Enable selective testing of integrations on main and backport branches

[jsoriano]

Running tests for all integrations after any merge launches dozens of jobs. If multiple branches
are merged in a short period of time, this can overload CI.

[v1v]

I've been thinking about this, and I wonder if the mapping one-one integration/worker could be changed to n-1 integrations/worker?

So the algorithm will split the number of integrations by X and run them sequentially.

Pros:

• Less number of requested workers
• Less wait for provisioners to provide workers

Cons:

• Builds might take a bit longer for merge commits

What do you think?

[mrodm]

👍
I think we can do this change, the more packages/integrations are developed, the more time would be spent here testing all integrations.

Applying this change, I think it could be good. Just testing those packages that are involved in the PR. IIRC there will be cases that all the packages will be tested. For instance when .ci/Jenkinsfile or links_table.yml are modified.

In those cases where all the integrations are going to be executed, we can do that n-1 algorithm to run in parallel the testing.

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

❕ Build Aborted

Either there was a build timeout or someone aborted the build.

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-11-21T09:05:06.460+0000

• Duration: 243 min 28 sec

Test stats 🧪

Test	Results
Failed	0
Passed	5155
Skipped	9
Total	5164

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (333/333)	💚
Files	97.651% (582/596)	👎 -0.039
Classes	97.651% (582/596)	👎 -0.039
Methods	91.098% (5710/6268)	👎 -0.087
Lines	91.634% (115645/126203)	👎 -0.071
Conditionals	100.0% (0/0)	💚

[v1v]

Cosmetic change, so default values are as they were previously.

[jsoriano]

So the algorithm will split the number of integrations by X and run them sequentially.

What do you think?

@v1v this sounds like a good idea too, but I think both things could be applied independently, both things bring benefits.

Do we already have some helper to do this scheduling? We would need to think about how to do this split efficiently.
We have integrations that take much more time than others to run, only splitting the full list of integrations will pair the more heavy ones with others, prolonging a lot the total time. I think that ideally, for better balance, we should have some kind of queue that sequentially assign integrations to idle workers. This would be similar to what jenkins does, but limiting the workers a parallel block can use at the same time (is there an option for that? this would be a low hanging fruit for this).

Cosmetic change, so default values are as they were previously.

Suggestions applied, thanks!

[jsoriano]

I think that ideally, for better balance, we should have some kind of queue that sequentially assign integrations to idle workers. This would be similar to what jenkins does, but limiting the workers a parallel block can use at the same time (is there an option for that? this would be a low hanging fruit for this).

This would be something like this https://issues.jenkins.io/browse/JENKINS-44085?focusedCommentId=373436&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-373436