Improve error message if multiple "test" annotations are present on a method

[ctapobep]

For the following test two descriptors will be created -- one for @ParameterizedTest and another for @Test:

@ParameterizedTest
@ValueSource(strings = "blah")
@Test
void blah(String blah) {}

The parameterized test passes, while the other one fails because JUnit Juiter cannot resolve the parameter:

org.junit.jupiter.api.extension.ParameterResolutionException: No ParameterResolver registered for parameter [java.lang.String arg0] in executable [void io.qala.datagen.junit5.Junit5ParameterizedTest.blah(java.lang.String)].

I don't know if it's a bug or a feature, but because I didn't know about this I spent an hour debugging JUnit. I assume many more people could step on this, so would be great to make the logic more obvious. Couple of options:

• Ignore @Test if some other test descriptor is created
• Make error message clearer: ... This can happen e.g. if 2 different Test annotations are put on one method.
• Use mechanisms for creating Test Descriptors similar to how ParameterResolver works: boolean supportsTest(...)

Related Issues

• Introduce error handling mechanism for validation and discovery errors #242

[marcphilipp]

Agreed, there should definitely be a better error message.

[sbrannen]

Note that this is likely a more general topic involving all types of "test" annotations such as @Test, @TestTemplate, and @TestFactory.

Keep in mind that @RepeatedTest and @ParameterizedTest are specializations of @TestTemplate.

I have therefore changed this issue's title accordingly.

[akozlova]

BTW I've wrote inspections for IntelliJ to highlight such dbl tests.

[sbrannen]

BTW I've wrote inspections for IntelliJ to highlight such dbl tests.

Nice, @akozlova! 👍

[sbrannen]

@akozlova, in IntelliJ do you simply inspect for multiple annotations meta-annotated with @Testable?

Or do you check for individual "test" annotations?

[akozlova]

Now it's only check for predefined annotations, mainly due to the fact that IDEA knows nothing about custom engines and thus can't check them: who knows, probably some engine would require both parameterized and some testable annotation due to the fact that jupiter won't be available/filtered.

[sbrannen]

Now it's only check for predefined annotations, mainly due to the fact that IDEA knows nothing about custom engines and thus can't check them:

Makes sense!

who knows, probably some engine would require both parameterized and some testable annotation due to the fact that jupiter won't be available/filtered.

Yeah, I was pondering the exact same thing. 😉

[smoyer64]

I accidentally ended up with @Test and @TestTemplate on the same test method (converting from a non-parameterized test to a parameterized test) and it actually ran both. When complete, there were five tests provided by the parameterized annotation and one test by the normal test annotation. Note that the non-parameterized test failed due to lack of parameter resolution.