jgiven interferes with testng failures handling mechanism

[adrian-herscu]

Using TestNG 6.11 with JGiven 0.14.

I am trying to implement a generic mechanism that repeats failing tests with predefined repetition and interval configuration.

In TestNG there is a retryAnalyzer attribute on the @Test annotation.

It appears that JGiven somehow catches AssertionErrors and does not pass them to TestNG correctly, hence the my attached RetryAnalyzer does not get invoked.

Here is an example code:

@Slf4j
@SuppressWarnings({ "javadoc", "nls", "static-method", "boxing", "hiding" })
public final class RetryTests
    extends
    ScenarioTest<RetryTests.SomeGiven<?>, RetryTests.SomeWhen<?>, RetryTests.SomeThen<?>> {

    @Slf4j
    public static final class MyAnalyzer extends RetryAnalyzerCount {
        public MyAnalyzer() {
            setCount(4);
        }

        @Override
        public boolean retryMethod(ITestResult result) {
            ThreadUtils.sleep(200);
            log.debug("{} failed, retrying count {}",
                result.getMethod().getQualifiedName(),
                getCount());
            return true;
        }
    }

    public static class SomeGiven<SELF extends SomeGiven<SELF>>
        extends Stage<SELF> {
        // empty
    }

    public static class SomeThen<SELF extends SomeThen<SELF>>
        extends Stage<SELF> {

        public SELF should_fail() {
            Assert.assertEquals("james", "JamesFail");
            return self();
        }
    }

    public static class SomeWhen<SELF extends SomeWhen<SELF>>
        extends Stage<SELF> {
        // empty
    }

    @Test(retryAnalyzer = MyAnalyzer.class)
    public void _1_plainTestNGTestIsRetried() {
        Assert.assertEquals("james", "JamesFail");
    }

    @Test(retryAnalyzer = MyAnalyzer.class)
    public void _2_jgivenTestIsNotRetriedBecauseAssertionErrorIsSwallowed() {
        then().should_fail();
    }

    @Test(retryAnalyzer = MyAnalyzer.class)
    public void _3_andThisOneShowsAssertionErrorIsSwallowedByJgiven() {
        try {
            then().should_fail();
        } catch (final Throwable t) {
            log.debug("throwable caught {}", t); // ISSUE: never happens
        }
    }
}

[janschaefer]

Yes. JGiven catches all exceptions. So number 3 is expected behavior. This is to be able to show steps as skipped that follow the failed step. I have to look deeper in this issue and see how this is related to the retryAnalyzer and why number 2 is not working as expected, because JGiven passes the exception to TestNG, but maybe there is some ordering issue and the retryAnalyzer is evaluated before JGiven passes the exception back to TestNG.

[p-brz]

I believe that this can be related to a possible bug that i found.
What ocurred with me is that a failing test with jgiven, despite being marked with the FAILURE status, was added to the "passed tests" in the testng context.
To solve this, i extended the ScenarioTestListener and overrided the onTestSuccess method:

@Override
public void onTestSuccess(ITestResult result) {
	super.onTestSuccess(result);

	if(result.getStatus() == ITestResult.FAILURE){
		result.getTestContext().getFailedTests().addResult(result, result.getMethod());
		result.getTestContext().getPassedTests().removeResult(result);
	}
}

I believe that it's not necessary to extend the ScenarioTestListener (a custom listener should work).

@adrian-herscu, maybe you can test this to see if this works for your problem.

In any case, i think this should be included in the ScenarioTestListener directly. So, @janschaefer, maybe you could include this into the code if you think this is ok.

[adrian-herscu-ge]

@p-brz tried your solution... unfortunatelly, no joy :(

Added @listeners({ScenarioTestListenerEx.class}) to the class above, added logs and got the following:

15:42:18.045 [DEBUG] [main ]: ScenarioTestListenerEx - test result was [TestResult name=_2_jgivenTestIsNotRetriedBecauseAssertionErrorIsSwallowed status=FAILURE method=JGivenSelfTests._2_jgivenTestIsNotRetriedBecauseAssertionErrorIsSwallowed()[pri:0, instance:JGivenSelfTests@587c290d] output={null}]
15:42:18.045 [TRACE] [main ]: ScenarioTestListenerEx - handling failure condition

So, everything seems hooked fine. Do I miss something?

[VPazzini]

I am having the same problem, any updates on how to work around it?

[adrian-herscu]

I barely remember what the problem was... From 50,000 feet... JGiven does not play nice with every TestNG feature. So, if you plan using retries, concurrency and/or dependencies, you should expect trouble.

…

On Thu, 11 Apr 2019 at 14:49, Vinícius S. Pazzini ***@***.***> wrote: I am having the same problem, any updates on how to work around it? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#312 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC25QXWmiV-E3H98Y7XIHtFEAOxe_sDDks5vfyFEgaJpZM4MsxDu> .

[VPazzini]

The problem is that jgiven intercepts all exceptions, so testng can't see that the test failed. Is there a way to make given not ok intercept those exceptions?

[janschaefer]

I decided to fix this issue by changing the behavior of JGiven to not catch exceptions in steps anymore in TestNG tests. See #422.