Run tests with Gradle test runner instead of randomizedtesting.junit4-ant

[alpar-t]

Currently we use https://github.com/randomizedtesting/ for running the tests.

The Gradle plugin calls into ant to do the heavy lifting.
It also replaces the Gradle Test task that the java plugin creates ( called test ) with it's own RandomizedTestingTask . This proved to be problematic, see #31324, and might no longer be possible in some future Gradle release.
Also, running the tests in parallel with the ant runner is independent from Gradle and does not honor Gradle --max-workers setting, which makes it hard to make other parts of Gradle run in parallel as it can lead to too many threads being created.

It is possible to use the reandmizedtesting runner i.e. @RunWith and run the tests with the Grade test runner. This proposal is to replace the ant part and not the runner.

[elasticmachine]

Pinging @elastic/es-core-infra

[alpar-t]

Looking at what the ant runner implements and what we use it seems that we'll loose test class randomization and heartbeat. The former might eventually land in junit5 junit-team/junit5#13.
Gradle also measures how long tests take, so we should be fine without the heart beat.

It doesn't seem extremely difficult to implement random test order in Gradle, there is a precedent for it:
https://github.com/gradle/gradle/blob/93c34bdb6709f7ae9276c28e3d1b47b0495abb6a/subprojects/testing-base/src/test/groovy/org/gradle/api/internal/tasks/testing/processors/RunPreviousFailedFirstTestClassProcessorTest.groovy

However this is internal API so would need to convince Gradle it's useful and contribute it.

Other features of randomizedtesting, like stall thread detection are implemented in the runner.

[rjernst]

it seems that we'll loose test class randomization and heartbeat.

I don't think it is acceptable to lose any of the functionality from randomized runner. We would need to implement this functionality over the gradle runner. I think it is all doable, though, at least with some work.

[alpar-t]

I agree. Especially the w.r.t the ordering of tests. Gradle doesn't have a public API (or any API I could find for that matter) that we can use. Implementing this by making changes to Gradle isn't neither hard or time consuming, but will have to figure out if that's something Gradle will accept, or if we are willing to live with the overhead of maintaining a custom Gradle distribution.

Another distinction that we might have to make is: are we ok to go without randomizing test class order temporarily, but I think in order to have an informed decision we would need to measure the advantages in terms of opportunity of build time reduction this will give us, and cluster formation improvements (#30904) also come into play there.

I think the first thing is to get to have a PoC run of the tests with Gradle so that we can at least have an initial measurement and comparison.

[alpar-t]

I isolated the hanging tests task in this repo: https://github.com/atorok/gradle_test_randomizedtesting_deadlock

[alpar-t]

The problem is in BootstrapForTesting if removed the tests run.

[alpar-t]

It's the security manage ./gradlew :server:test -Dtests.security.manager=false doesn't block and there's already gradle/gradle#3526

[alpar-t]

The Gradle runner seems to be a bit slower, running :server:test in 2m 6s vs 1m 52s (12.5%) both running on 6 forks, average of 3 runs each.

[rjernst]

Another distinction that we might have to make is: are we ok to go without randomizing test class order temporarily

No, losing this would be a deal breaker. Reproducibility is paramount above all other concerns. It doesn't matter if we could run all tests in 10 seconds; if we can't reproduce failures, the speed is worthless.

It is unfortunate (but not surprising) that gradle cannot handle running with security manager. This is also a deal breaker for using the gradle runner.

Another option I considered which would gain us speed, would be to have separate randomized runner tasks, one for each jvm we plan to spawn. We could then do the assignment of classes to jvms late in configuration (or just before execution in a common doFirst or dependsOn). This would allow us all the utility of randomized runner, but utilizing separate tasks would allow parallelization with gradle.