Running @QuarkusTest in Parallel

[bcluap]

Is it possible to split ones tests across multiple classes and run them in parallel against one instance of Quarkus?

I've managed to get JUnit ordering working and can run test classes in an order I want, but many of the tests take a long time (they are integration tests and call into Quarkus that then calls into external APIs). So lets say I have 10 test classes annotated with @QuarkusTest - and each one takes say 2min to run. Right now they run one after the other and takes 20 minutes. Can I possibly run all 10 in parallel and hence taking more like 2min (the tests are not CPU bound).

If I use this config:
junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.classes.default=concurrent
junit.jupiter.execution.parallel.mode.default=same_thread

Then JUnit tries to start up multiple instances of Quarkus as opposed to using once instance and running the tests in parallel.

[turf00]

I also encountered the same problem and was wondering what the general suggested solution is.

[roggenbrot]

Since I'm trying to implement parallelism for the quarkus-spock extension at the moment I would be interested in the answer as well.

Looking at the source code of the QuarkusTestExtension the quarkus application is a static variable

quarkus/test-framework/junit5/src/main/java/io/quarkus/test/junit/QuarkusTestExtension.java

Line 125 in e47a267

private static RunningQuarkusApplication runningQuarkusApplication;

so as far as the test execution is not forked (not a junit5 expert... ) or a custom classloading mechanism is used for each test class (seems not to be supported in junit 5 junit-team/junit5#201) my guess is that running quarkus test specifications in parallel is not supported at the moment.

Could someone confirm/deny this guess?

[roggenbrot]

What you can do is enabling forking in surefire:

<plugin>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>${surefire-plugin.version}</version>
      <configuration>
          <forkCount>4</forkCount>
          <reuseForks>false</reuseForks>
          <systemPropertyVariables>
              <java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
              <maven.home>${maven.home}</maven.home>
          </systemPropertyVariables>
      </configuration>
  </plugin>

Make sure that you've set you ports config to dynamic ports (e.g. quarkus.http.test-port=0)

[bcluap]

I did a workaround that solves for my use case. It allows one to have many test classes running in parallel and all test against the same running instance of Quarkus. Our tests went from 15 minutes to 1 minute. No need to have lots of Quarkus instances running. The steps are a bit hackey but work perfectly. The just of it is that Quarkus kicks off a test that in turn uses the JUnit libraries to kick off your actual tests in parallel:

1. junit-platform.properties in src/test/resources contains something along the lines of:

junit.jupiter.execution.parallel.config.strategy=fixed
junit.jupiter.execution.parallel.config.fixed.parallelism=5
junit.jupiter.execution.parallel.enabled=true
junit.jupiter.execution.parallel.mode.classes.default=concurrent
junit.jupiter.execution.parallel.mode.default=concurrent
junit.jupiter.testclass.order.default=org.junit.jupiter.api.ClassOrderer$OrderAnnotation
junit.jupiter.testmethod.order.default=org.junit.jupiter.api.MethodOrderer$OrderAnnotation

2. Have a single "driver" test class annotated with @QuarkusTest This class is all Quarkus sees as the integration test but it in turn kicks off the actual tests. Here is my source:

@QuarkusTest
public class DriverTest {

    private static final Logger log = LoggerFactory.getLogger(EclipseTest.class);
    private boolean testsFinished = false;

   

    @Test
    @Order(1)
    void startup() {
        log.info("Starting Quarkus to run integration tests");
        // Do something to know Quarkus has started. You could just sleep if you want
        log.info("Quarkus has started. Now running tests");
        int failures = runAllTests();
        assertThat(failures).isZero();
    }

    public int runAllTests() {
        SummaryGeneratingListener summaryListener = new SummaryGeneratingListener();
        String filter = ".*";
        log.info("Running all tests with class name matching [{}]", filter);
        LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
                .selectors(selectPackage("com.ukheshe.eclipse.rest.tests"))
                .filters(includeClassNamePatterns(filter))
                .build();
        Launcher launcher = LauncherFactory.create();
        launcher.execute(request, summaryListener, new TestExecutionListener() {
            @Override
            public void testPlanExecutionFinished(TestPlan testPlan) {
                log.info("Test plan is finished");
                testsFinished = true;
            }

            @Override
            public void testPlanExecutionStarted(TestPlan testPlan) {
                log.info("\n\n\n######################################################\n"
                        + "We will be running [{}] tests\n"
                        + "######################################################\n\n\n",
                        summaryListener.getSummary().getTestsFoundCount());
            }
        });
        while (!testsFinished) {
            Thread.currentThread.sleep(1000);
        }
        System.out.println("######################################################");
        summaryListener.getSummary().printFailuresTo(new PrintWriter(System.out), 0);
        summaryListener.getSummary().printTo(new PrintWriter(System.out));
        System.out.println("######################################################");

        
        return summaryListener.getSummary().getFailures().size();
    }
}

3. Your actual unit test classes can just be annotated with normal Junit @test annotation e.g.

public class TestFunctionalArea1  {
    @Test
    void someTest() {
...

[mkouba]

Is it possible to split ones tests across multiple classes and run them in parallel against one instance of Quarkus?

I don't think that this is officialy supported and could also cause troubles when using testing profiles.

But @geoand @aloubyansky @gsmet and others will know more for sure ;-)

[geoand]

Currently, it will not work properly

[franden]

we have the same requirement for parallel execution of @QuarkusTest test classes. is there any chance that this will be supported in near future?

[ymajoros]

Any news on this?

[geoand]

Nope