Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Brainstorming: How to handle consecutive steps scenario #166

Closed
nikowitt opened this issue Nov 11, 2015 · 12 comments
Closed

Brainstorming: How to handle consecutive steps scenario #166

nikowitt opened this issue Nov 11, 2015 · 12 comments
Milestone

Comments

@nikowitt
Copy link
Contributor

Hi Jan,

currently, I'm trying to create a table based scenario that depends on consecutive steps:

    // virtual "dataprovider" as we want don't want to execute an own scenario for each record, but to execute it consecutively
    private static Object[][] create_correspondence_with_reference_number_category_provider() {
        return $$(
                $(CODE, CODE, null, 1),
                $(CODE, CODE, null, 2),
                $(CODE, CODE, CATEGORY_NAME_1, 1),
                $(CODE, CODE, null, 3),
                $(CODE, CODE, CATEGORY_NAME_1, 2),
                $(CODE, CODE, CATEGORY_NAME_2, 1),
                $(CODE, CODE, CATEGORY_NAME_1, 3),
                $(CODE, CODE, CATEGORY_NAME_2, 2));
    }

    @Test
    public void create_correspondence_with_reference_number_category() throws Exception {

        given().logged_in_system_user().and().a_new_correspondence_code(CODE).and()
                .new_reference_number_categories(CATEGORY_NAME_1, CATEGORY_NAME_2).and().a_refno_template_with_category();

        assert_that().sequence_is_correct_when_correspondence_with_reference_is_created_consecutively(
                create_correspondence_with_reference_number_category_provider());

    }

(...)
    public void sequence_is_correct_when_correspondence_with_reference_is_created_consecutively(@Table(columnTitles = { "sender code",
            "recipient code",
            "reference category",
            "expected sequence" }) Object[][] data)
            throws Exception {
        for (Object[] o : data) {
            reference_with_sender_$_recipient_$_is_created((String) o[0], (String) o[1], (String) o[2]);
            reference_sequential_number_is((int) o[3]);
        }

    }

When everything is fine and no errors are raised, I can create the following output:

Test Class: com.sobis.pirsjava.tests.businessobject.BOReferenceNumberTest

 Scenario: create correspondence with reference number category

         Given logged in system user
           And a new correspondence code
           And new reference number categories category1, category2
           And a refno template with category
   Assert that sequence is correct when correspondence with reference is created consecutively

     | sender code | recipient code | reference category | expected sequence |
     +-------------+----------------+--------------------+-------------------+
     | NEWCODE     | NEWCODE        | null               |                 1 |
     | NEWCODE     | NEWCODE        | null               |                 2 |
     | NEWCODE     | NEWCODE        | category1          |                 1 |
     | NEWCODE     | NEWCODE        | null               |                 3 |
     | NEWCODE     | NEWCODE        | category1          |                 2 |
     | NEWCODE     | NEWCODE        | category2          |                 1 |
     | NEWCODE     | NEWCODE        | category1          |                 3 |
     | NEWCODE     | NEWCODE        | category2          |                 2 |

This is a much better approach than to create a when/then for every case as the scenario grows pretty fast as soon as one additional case is added.

But of course, this way, an error case cannot be displayed in the table itself. To do that, I'd have to be able to catch errors, mark the row where an error occurs and then throw a global error afterwards.

An error case currently looks like this

Test Class: com.sobis.pirsjava.tests.businessobject.BOReferenceNumberTest

 Scenario: create correspondence with reference number category

         Given logged in system user
           And a new correspondence code
           And new reference number categories category1, category2
           And a refno template with category
   Assert that sequence is correct when correspondence with reference is created consecutively

     | sender code | recipient code | reference category | expected sequence |
     +-------------+----------------+--------------------+-------------------+
     | NEWCODE     | NEWCODE        | null               |                 1 |
     | NEWCODE     | NEWCODE        | null               |                 2 |
     | NEWCODE     | NEWCODE        | category1          |                 1 |
     | NEWCODE     | NEWCODE        | null               |                 3 |
     | NEWCODE     | NEWCODE        | category1          |                 2 |
     | NEWCODE     | NEWCODE        | category2          |                 1 |
     | NEWCODE     | NEWCODE        | category1          |                 3 |
     | NEWCODE     | NEWCODE        | category2          |                22 |

FAILED: 
Expected: is <22>
     but: was <2>

It is still readable, But only when the assert fails on a unique sequence so I can tell from which row the error is raised without debugging.

Any idea? :)

Best regards,
Niko

@janschaefer
Copy link
Contributor

A simple solution would be to add an index to the rows and add a custom error message to your assertion message. Then at least you should be able to correlate the error with the row.

@nikowitt
Copy link
Contributor Author

Oh yes, you are right :)

@nikowitt
Copy link
Contributor Author

Interesting, adding "numberedRows"=true throws an exception in my case:

java.lang.UnsupportedOperationException
    at java.util.AbstractList.add(Unknown Source)
    at com.tngtech.jgiven.report.model.DataTable.addColumn(DataTable.java:85)
    at com.tngtech.jgiven.report.model.StepFormatter.addNumberedRows(StepFormatter.java:206)
    at com.tngtech.jgiven.report.model.StepFormatter.toTableValue(StepFormatter.java:189)
    at com.tngtech.jgiven.report.model.StepFormatter.getRemainingArguments(StepFormatter.java:179)
    at com.tngtech.jgiven.report.model.StepFormatter.buildFormattedWordsInternal(StepFormatter.java:145)
    at com.tngtech.jgiven.report.model.StepFormatter.buildFormattedWords(StepFormatter.java:98)
    at com.tngtech.jgiven.impl.ScenarioModelBuilder.createStepModel(ScenarioModelBuilder.java:102)
    at com.tngtech.jgiven.impl.ScenarioModelBuilder.addStepMethod(ScenarioModelBuilder.java:84)
    at com.tngtech.jgiven.impl.ScenarioModelBuilder.stepMethodInvoked(ScenarioModelBuilder.java:215)
    at com.tngtech.jgiven.impl.StandaloneScenarioExecutor$MethodHandler.handleMethod(StandaloneScenarioExecutor.java:127)
    at com.tngtech.jgiven.impl.intercept.StepMethodInterceptor.doIntercept(StepMethodInterceptor.java:57)
    at com.tngtech.jgiven.impl.intercept.StandaloneStepMethodInterceptor.intercept(StandaloneStepMethodInterceptor.java:30)
    at com.sobis.pirsjava.tests.businessobject.BOReferenceNumberTest$BOReferenceNumberStage$$EnhancerByCGLIB$$e5a1d644.sequence_is_correct_when_correspondence_with_reference_is_created_consecutively(<generated>)
    at com.sobis.pirsjava.tests.businessobject.BOReferenceNumberTest.create_correspondence_with_reference_number_category(BOReferenceNumberTest.java:83)
(...)org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:174)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

This is probably related to my Object[][]?

@janschaefer
Copy link
Contributor

Oh. That can be regarded as a defect :-(

@janschaefer janschaefer added this to the v0.9.5 milestone Nov 11, 2015
@nikowitt
Copy link
Contributor Author

I've encountered another issue: I'm using assertions that are not thrown and cannot be intercepted properly in the loop. My idea then is to set a flag on the Object[][](just add another column), but it seems that changes on the object that are done in the method itself are not reflected when the object is formatted. Is this a technical limitation for some reason?

@janschaefer
Copy link
Contributor

That is a limitation, because the table is copied internally when the step is executed. I think this is not possible to change.

@nikowitt
Copy link
Contributor Author

Oh, what a pity. Then I have to check how to catch the assertions properly.

@janschaefer
Copy link
Contributor

A possible solution would be that you print the whole table again when an assertion fails. Then you can mark the corresponding line.

@nikowitt
Copy link
Contributor Author

One more question related to that topic: in a test step, I sometimes call further test steps that also could be invoked directly via given(), when(), then(). Is my assumption correct that in these cases, exceptions are not thrown, but the interceptor instead stops the execution directly?

@janschaefer
Copy link
Contributor

There should actually be no difference how you invoke the steps. In any case the exception is first captured by the interceptor, then the following steps are executed in a special skip mode and at the end the captured exception is thrown.

@nikowitt
Copy link
Contributor Author

Let me explain my issue with more details:

In addition to the intial post:

public void sequence_is_correct_when_correspondence_with_reference_is_created_consecutively(
                @Table(numberedRows = true, columnTitles = { "sender code",
                        "recipient code",
                        "reference category",
                        "expected sequence" }) List<Object[]> data)
                throws Exception {

            for (int i = 0; i < data.size(); i++) {
                Object[] o = data.get(i);

                try {
                    reference_with_sender_$_recipient_$_is_created((String) o[0], (String) o[1], (String) o[2]);
                    reference_sequential_number_is((int) o[3]);
                } catch (Throwable e) {
                    throw new TestStageException("Error in row " + i);
                }

            }

        }

This is the original approach that does not work.

Checking more closely:

    public BOReferenceNumberStage reference_sequential_number_is(int i) throws Exception {
            field_$_of_$_$_match(ReferenceNumber.SEQUENTIALCODE, correspondence.getRefNo(), true, is(i));
            return self();
        }

This method is actually only a wrapper for a technical test. The method is also directly usable in a technical then step.

public T field_$_of_$_$_match(@Quoted String field, IDatabaseEntity entity, @IsIsNot boolean matches,
            Matcher<?>... matcher)
            throws Exception {
        Object value = ReflectionUtils.getRelatedObjectByFieldName(entity, field);

        for (Matcher<?> single : matcher) {
            Matcher<Object> m = (Matcher<Object>) (matches ? single : not(single));
            LOGGER.debug("MATCHER={} on object {}", m, value);
            assertThat(value, m);

        }

        return self();
    }

The assertThat failure is not thrown, but intercepted as explained by you. When I directly replace the technical matcher with the assert, it works as expected:

public void sequence_is_correct_when_correspondence_with_reference_is_created_consecutively(...)
(...)
        try {
                    reference_with_sender_$_recipient_$_is_created((String) o[0], (String) o[1], (String) o[2]);
                    assertThat(correspondence.getRefNo().getSequentialCode(), is((int) o[3]));
                } catch (Throwable e) {
                    throw new TestStageException("Error in row " + i);
                }

In collection handling checks, this also causes issues when I want to catch/collect failures from another step method. So at least for me the default approach of intercepting all exceptions is not in some corner cases.

Maybe some mechanism to tell the interceptor to further throw an exception instead of intercepting it can be introduced. Maybe some global flag can be applied to pass exceptions when I'm in a batch mode? E.g. something like

try {
                    reference_with_sender_$_recipient_$_is_created((String) o[0], (String) o[1], (String) o[2]);
                    deactivateInterception();
                    reference_sequential_number_is((int) o[3]);
                    activateInterception();
                } catch (Throwable e) {
                    throw new TestStageException("Error in row " + i,e);
                }

@janschaefer
Copy link
Contributor

I have to check the code, but as far as I know, exceptions should only be captured on the outer calls of step methods and not on the inner ones. If this is not the case, this can be easily fixed. Which also would totally make sense.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants