Cucumber custom config strategy with exclusive isolated resources not running the scenarios in parallel

[iamsethi]

👓 What did you see?

Given I am using below custom config strategy to run test in parallel e.g junit-platform.properties

cucumber.execution.parallel.config.strategy=custom
cucumber.execution.parallel.config.custom.parallelism=2
cucumber.execution.parallel.config.custom.class=com.runners.CustomStrategy

Then scenarios under Feature1,Feature2 and Feature3 are running parallelly without any issues

When i add @isolated tag to my Feature2 and update my junit-platform.properties

cucumber.publish.quiet=true
cucumber.execution.parallel.enabled=true
cucumber.filter.tags=@pricePackArchitectureV2
cucumber.execution.parallel.config.strategy=custom
cucumber.execution.parallel.config.custom.parallelism=2
cucumber.execution.parallel.config.custom.class=com.runners.CustomStrategy
cucumber.plugin=pretty,json:results/cucumber.json,junit:results/cucumber.xml
cucumber.execution.exclusive-resources.isolated.read-write=org.junit.platform.engine.support.hierarchical.ExclusiveResource.GLOBAL_KEY

✅ What did you expect to see?

Then I am expecting all the scenarios except with Feature2(@isolated tag) should run in parallel
But the scenarios under Feature1 and Feature2 are also started running sequentially

📦 Which tool/library version are you using?

cucumber : 7.6.0
openjdk 21.0.2 2024-01-16
OpenJDK Runtime Environment (build 21.0.2+13-58)
OpenJDK 64-Bit Server VM (build 21.0.2+13-58, mixed mode, sharing)

🔬 How could we reproduce it?

📚 Any additional context?

CustomStrategy class

import org.junit.platform.commons.JUnitException;
import org.junit.platform.engine.ConfigurationParameters;
import org.junit.platform.engine.support.hierarchical.ParallelExecutionConfiguration;
import org.junit.platform.engine.support.hierarchical.ParallelExecutionConfigurationStrategy;

import java.util.concurrent.ForkJoinPool;
import java.util.function.Predicate;

public class CustomStrategy implements ParallelExecutionConfiguration, ParallelExecutionConfigurationStrategy {
    private int parallelism;

    @Override
    public int getParallelism() {
        return parallelism;
    }

    @Override
    public int getMinimumRunnable() {
        return 0;
    }

    @Override
    public int getMaxPoolSize() {
        return 256 + parallelism;
    }

    @Override
    public int getCorePoolSize() {
        return parallelism;
    }

    @Override
    public int getKeepAliveSeconds() {
        return 60;
    }

    @Override
    public Predicate<? super ForkJoinPool> getSaturatePredicate() {
        return (ForkJoinPool p) -> true;
    }

    @Override
    public ParallelExecutionConfiguration createConfiguration(ConfigurationParameters configurationParameters) {
        this.parallelism = configurationParameters.get("custom.parallelism", Integer::valueOf).orElseThrow(() ->
                new JUnitException(String.format("Configuration parameter '%s' must be: ",
                        "junit.jupiter.execution.parallel.config.custom.parallelism")));
        return this;
    }
}

[mpkorstanje]

Are you running a single feature and seeing all scenarios in that feature run in sequence? And if you are running multiple features, only the features with isolated scenarios run their scenarios sequentially?

If so, then it looks like you are running into junit-team/junit-framework#2982. Duplicate of #2589.

There isn't much we can do in Cucumber, but you may want to propose a fix in JUnit.

[iamsethi]

@mpkorstanje I am running multiple feature and none of them are running in parallel

Feature1
Feature2 - tagged with @isolated tags
Feature3

[mpkorstanje]

I can't reproduce the behavior where all scenarios are execute serialy. Do you have a more precise minimal reproducer? You could fork the cucumber-java-skeleton and use that as a base.

[iamsethi]

@mpkorstanje Please find the sample attached
selenium-springboot.zip

PS:
The project is using Java 17
I've commented line 11 under src/test/resources/junit-platform.properties. Please uncomment that to test the issue
Run the test : mvn clean test -Dcucumber.filter.tags="@Regression"

[mpkorstanje]

Unfortunately your reproducer is rather large and includes many parts that do not appear to be relevant to your problem. For future reproducers please ensure they only include the bare necessities needed to reproduce the problem.

And because the reproducer depends on Selenium or Selenium Grid I could not also not execute the reproducer without replacing all Selenium interactions with Thread.sleep. Then logging the features uri, scenario name and thread shows that @isolated scenarios run all on the same thread (as expected) and the the regular scenarios run concurrently on different threads.

    @Before
    public void before(Scenario scenario) {
        System.out.println(scenario.getUri() + " " + scenario.getName() + " " + Thread.currentThread() );
    }

So I can not reproduce your problem. Can you simplify your reproducer and verify that Selenium is essential to reproducing the problem? You will also want to pay attention to the following, while unlikely to cause the problem, these aspects make debugging harder:

1. Use up to date versions of Cucumber, JUnit and Spring.
2. Use the cucumber-bom and the junit-bom, this also removes the need to set versions for these dependencies.
3. Remove unused dependencies such as testng and cucumber-testng, junit-platform-launcher, ect.
4. Remove transitive dependencies already provided by spring-boot-starter-test such as junit-jupiter-engine,
5. Feature files should go under src/test/resources/com/example/app and can then be selected with @SelectPackage("com.example.app").
6. Configure Surefire to use cucumber.junit-platform.naming-strategy=long to disambiguate between examples. See https://github.com/cucumber/cucumber-java-skeleton/blob/main/maven/pom.xml#L84 and https://github.com/cucumber/cucumber-jvm/tree/main/cucumber-junit-platform-engine#surefire-and-gradle-workarounds.

[mpkorstanje]

It doesn't look like the recording got uploaded properly.

[mpkorstanje]

The current reproducer does reproduce the problem.

With 3 feature files and 3 threads, the two isolated features end up waiting to acquire the global read write lock. This leaves only 1 thread available to work on the remaining scenarios that are not isolated. So this is indeed distinct from junit-team/junit-framework#2982 and I think we'd have to create a new issue there.

As a work around you could consider:

• using two runner classes. One for the isolated and one for the regular scenarios.
• increasing the parallelism, and using the resource pool pattern to ensure only a limited number of web drivers used at the same time. You can use Apache Commons Pool to facilitate that.

[mpkorstanje]

@iamsethi I've deleted your post with the reproducer. I just noticed you left some credentials in application-dev.properties. That was probably not meant to be shared.

[iamsethi]

@mpkorstanje Removed application-dev.properties. Uploading it again so that anyone can use this as a POC for Cucumber + JUnit 5 + Spring Boot by searching for this issue to replicate this

selenium-springboot.zip

Video :
https://github.com/user-attachments/assets/6bf6c699-e10d-46e1-a9b2-f2e47336e4e1

[mpkorstanje]

Cheers. I made a pure JUnit Jupiter version of your reproducer and created an issue for the JUnit team. Any fix is likely to come from there so you'll want to follow that.

[fslev]

Issue should be fixed by Junit 5.11.1 recently released. It also brings other parallelism fixes.
Maybe cucumber-jvm should also be updated with the new dependency and released.

[mpkorstanje]

If you're using JUnit 5 in the recommended manner, you only have to upgrade JUnit 5. The bill of materials will straighten out the dependencies.

• https://junit.org/junit5/docs/current/user-guide/#running-tests-build-gradle-bom
• https://junit.org/junit5/docs/current/user-guide/#running-tests-build-maven-bom