JGiven parallelization issues

[l-1squared]

supersedes Issue #300, #823 and Issue #491

Tasks:

• check why the Parallel Methods Tests from testng parallel methods mode failures #491 fails when running in parallel with TestNG ✅

• understand TestNG test lifecycle (is it per method or per class) ✅ (test methods share their class instance)

• check why the adjacent MultiThreadingTest does not fail. ✅

• check why the same test appears to succeed when running it in parallel with maven ✅ (JUnit, different lifecycle)

• write tests for parallel execution checks for JUnit ✅

• write tests for parallel execution checks for JUnit5 ✅

• fix concurrency issue that occurs when multiple threads merge scenario cases ✅(PR Bugfix/issue 829 junit5 error #842)

• fix injected stages get used for multiple scenarios with TestNG (PR Bugfix/issue 829 testng multithread #878)

• remove Maven Mojo Threadsafe marker if not appropriate ❌

• credit Adrian-Herscu for filing a bug if appropriate ✅

• credit Hatzen for thread safe addition if appropriate✅

[l-1squared]

First observations:

• Error occurs with injected Sceanrio stages only. Adrians test works fine if we restrict ourselves to inherited stages, yet it already fails, if we use injected stages only.
• Creating a similar test for JUnit5 works just fine, at least if the tests run in a per_method lifecycle, per class lifecycle could not be tested, because even single thread tests fail doing so.

[l-1squared]

The core of the issue with TestNG tests seems to be that TestNG uses the same instance for both tests that are run in parallel. This turns out to not be an issue if one uses the stages provided by the generic declaration, because those refer to the thread dependent scenario and hence we get individual stages for each thread. In case of the injected stages however, the second thread to pass by "Inject Stages" will overwrite the first one. Now both threads operate on the same scenario and consequently mess up each others state. JGiven expects a single scenario to be on a single thread, The reuse of the test class in testNG violates that principle, hence the error.
I will try to find a fix. @adrian-herscu

[l-1squared]

Found another concurrency issue that occurrs when we merge scenario cases that were evaluated in parallel into one scenario model.
For JUnit5 (others not tested) it is possible that another thread writes into the case array when we try to iteratively evaluate it.
This happens, because we when finalizing a scenario we choose a "lead thread" whose scenario model is picked to be the scenario model attached in the report.

[adrian-herscu]

Saw all this and believed that 1.3.0 fixes all multi-threading issues with TestNG integration...

Are parallel data-providers supported?

public class JGivenTestNgParallelDataProviderTests extends
    ScenarioTest<JGivenTestNgParallelDataProviderTests.SomeGiven<?>, JGivenTestNgParallelDataProviderTests.SomeWhen<?>, JGivenTestNgParallelDataProviderTests.SomeThen<?>> {

    public static class OtherGiven<SELF extends OtherGiven<SELF>>
        extends Stage<SELF> {
        @ProvidedScenarioState
        protected final ThreadLocal<Integer> aValue = new ThreadLocal<>();

        public SELF with(final Integer value) {
            aValue.set(value);
            return self();
        }
    }

    public static class OtherThen<SELF extends OtherThen<SELF>>
        extends Stage<SELF> {
        @ExpectedScenarioState
        protected ThreadLocal<Integer> aValue;

        public SELF the_value(final Matcher<Integer> rule) {
            // ISSUE: when running with enough parallelism aValue is null
            assertThat(aValue.get(), rule);
            return self();
        }
    }

    public static class OtherWhen<SELF extends OtherWhen<SELF>>
        extends Stage<SELF> {
        // empty
    }

    public static class SomeGiven<SELF extends SomeGiven<SELF>>
        extends Stage<SELF> {
        // empty
    }

    public static class SomeThen<SELF extends SomeThen<SELF>>
        extends Stage<SELF> {

    }

    public static class SomeWhen<SELF extends SomeWhen<SELF>>
        extends Stage<SELF> {
        // empty
    }

    @ScenarioStage
    protected OtherGiven<?> otherGiven;
    @ScenarioStage
    protected OtherThen<?>  otherThen;
    @ScenarioStage
    protected OtherWhen<?>  otherWhen;

    @DataProvider(parallel = true)
    private static Iterator<Integer> data() {
        // ISSUE up to 10 it seems working... with 100 it does not
        return IntStream.range(0, 100).iterator();
    }

    @Test(dataProvider = INTERNAL_DATA_PROVIDER)
    public void shouldRunWithParallelDataProvider(final Integer aValue) {
        otherGiven.with(aValue);
        otherThen.the_value(is(aValue));
    }
}

[l-1squared]

So, in principle, paralellization should work, albeit with certain limitations for TestNG. As these are not were not simple to fix, I opted to print a nice error, telling you about the problem...

[adrian-herscu]

So, in principle, paralellization should work, albeit with certain limitations for TestNG. As these are not were not simple to fix, I opted to print a nice error, telling you about the problem...

It would be great to have these limitations documented in the user manual.

[l-1squared]

Fair point