Introduce validation rules for tests

[jlink]

Status Quo

Using JUnit 5 annotations and mechanisms one can do a lot of things that are syntactically possible but don't make sense. For example, one can annotate a method with both @Test and @BeforeEach.

Proposal

Introduce a mechanism where JUnit itself but also other parties can register rules that will be checked during discovery time, e.g.:

• AmbiguousAnnotationsRule
• DuplicateExtensionsRule
• ProhibitNestedTestsRule (in case a project wants to forbid this style of testing)

Those rules should be listable and configurable from the launcher API: Error, Warning, Off

Rules could work on different level, e.g whole test plan, engine, package, class, method, etc.

Related Issues

• Introduce error handling mechanism for validation and discovery errors #242

[sbrannen]

This is definitely an interesting proposal.

Even if it's not a first-class extension point, we will likely need to implement certain validation rules in any case (like the @Test + @BeforeEach combination).

[sbrannen]

When you say "checked," what exactly do you mean?

Would you envision that JUnit log a warning? Or maybe throw an exception?

[jlink]

Should be configurable via launcher API. Some teams might want to prohibit certain things, others might just like to be warned.

Am 17.01.2016 um 23:20 schrieb Sam Brannen notifications@github.com:

When you say "checked," what exactly do you mean?

Would you envision that JUnit log a warning? Or maybe throw an exception?

—
Reply to this email directly or view it on GitHub.

[neilmce]

Issue #242 proposes collecting such validation errors during discovery & reporting them all at test execution time.
An alternative, possibly complementary, approach could be to use a new Java annotation processor within JUnit.

Do the JUnit team have any thoughts on using an annotation processor to produce compilation errors/warnings for ill-configured test classes?

Pros:

• full failure information given to the user at compilation time.
• automatic integration with IDE code navigation as these would be 'real' compiler warnings/errors.

Cons:

• The warnings/errors would only appear if Java annotation processing was enabled.
• Introduce error handling mechanism for validation and discovery errors #242 mentions a need for user configurability within the validation rules. This could be trickier with an annotation processor than it would be with runtime validation during discovery.

I've got half a mind to start playing with a proof of concept. I don't want to start if the JUnit team feel that annotation processors are a no go.

[marcphilipp]

Thanks for the proposal! We discussed it in the team today. We think an annotation processor is too limited to detect all the problems described in #242 and thus think that's we should rather implement a more general mechanism.