JUnit Foundation fails with NullPointerException in case of 'Theories' feature usage

[HardNorth]

Despite the fact that it's marked as 'experimental' 'Theories' framework is a pretty old feature. I can track it up to JUnit 4.6, the earliest version available on github. But if I try to run even a simplest test with @Theory annotation JUnit Foundation fails with a NullPointerException:

java.lang.NullPointerException
	at com.nordstrom.automation.junit.TimeoutUtils.applyTestTimeout(TimeoutUtils.java:93)
	at com.nordstrom.automation.junit.CreateTest.intercept(CreateTest.java:38)
	at org.junit.experimental.theories.Theories$TheoryAnchor$1.createTest(Theories.java)
	at org.junit.runners.BlockJUnit4ClassRunner.createTest$original$16gso2gv(BlockJUnit4ClassRunner.java:260)
	at org.junit.runners.BlockJUnit4ClassRunner.createTest$original$16gso2gv$accessor$nt1WBOeA(BlockJUnit4ClassRunner.java)
	at org.junit.runners.BlockJUnit4ClassRunner$auxiliary$2HEKf0Uj.call(Unknown Source)
	at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:443)
	at com.nordstrom.automation.junit.CreateTest.intercept(CreateTest.java:36)
	at org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java)
	at org.junit.experimental.theories.Theories$TheoryAnchor$1.createTest$accessor$jNPCaVGY(Theories.java)
	at org.junit.experimental.theories.Theories$TheoryAnchor$1$auxiliary$OKIv9xhR.call(Unknown Source)
	at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:443)
	at com.nordstrom.automation.junit.CreateTest.intercept(CreateTest.java:36)
	at org.junit.experimental.theories.Theories$TheoryAnchor$1.createTest(Theories.java)
	at org.junit.runners.BlockJUnit4ClassRunner$2.runReflectiveCall$original$qdrjh9Tv(BlockJUnit4ClassRunner.java:309)
	at org.junit.runners.BlockJUnit4ClassRunner$2.runReflectiveCall$original$qdrjh9Tv$accessor$3FlV2mmC(BlockJUnit4ClassRunner.java)
	at org.junit.runners.BlockJUnit4ClassRunner$2$auxiliary$xXc0fpDh.call(Unknown Source)
	at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:443)
	at com.nordstrom.automation.junit.RunReflectiveCall.intercept(RunReflectiveCall.java:85)
	at org.junit.runners.BlockJUnit4ClassRunner$2.runReflectiveCall(BlockJUnit4ClassRunner.java)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:306)
	at org.junit.experimental.theories.Theories$TheoryAnchor$1.methodBlock(Theories.java:232)
	at org.junit.experimental.theories.Theories$TheoryAnchor.runWithCompleteAssignment(Theories.java:223)
	at org.junit.experimental.theories.Theories$TheoryAnchor.runWithAssignment(Theories.java:209)
	at org.junit.experimental.theories.Theories$TheoryAnchor.evaluate(Theories.java:192)
	at com.nordstrom.automation.junit.RetryHandler.runChildWithRetry(RetryHandler.java:54)
	at com.nordstrom.automation.junit.RunChild.intercept(RunChild.java:59)
	at org.junit.experimental.theories.Theories.runChild(Theories.java)
	at org.junit.experimental.theories.Theories.runChild(Theories.java)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run$original$KDZK6cAR(ParentRunner.java:413)
	at org.junit.runners.ParentRunner.run$original$KDZK6cAR$accessor$7H1m3s9a(ParentRunner.java)
	at org.junit.runners.ParentRunner$auxiliary$solXL5h8.call(Unknown Source)
	at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:443)
	at com.nordstrom.automation.junit.Run.intercept(Run.java:55)
	at org.junit.runners.ParentRunner.run(ParentRunner.java)
	at org.junit.runners.BlockJUnit4ClassRunner.run$accessor$nt1WBOeA(BlockJUnit4ClassRunner.java)
	at org.junit.runners.BlockJUnit4ClassRunner$auxiliary$D8nLJ3YO.call(Unknown Source)
	at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:443)
	at com.nordstrom.automation.junit.Run.intercept(Run.java:55)
	at org.junit.runners.BlockJUnit4ClassRunner.run(BlockJUnit4ClassRunner.java)
	at org.junit.experimental.theories.Theories.run$accessor$N0MIaorS(Theories.java)
	at org.junit.experimental.theories.Theories$auxiliary$I820y2np.call(Unknown Source)
	at com.nordstrom.automation.junit.LifecycleHooks.callProxy(LifecycleHooks.java:443)
	at com.nordstrom.automation.junit.Run.intercept(Run.java:55)
	at org.junit.experimental.theories.Theories.run(Theories.java)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)

I've prepared a pack of tests to reproduce the issue, they are located here: https://github.com/reportportal/agent-java-junit/tree/develop/src/test/java/com/epam/reportportal/junit/features/theory

Here is an example of a simple test which fails:

import org.junit.Assume;
import org.junit.experimental.theories.Theories;
import org.junit.experimental.theories.Theory;
import org.junit.runner.RunWith;

@RunWith(Theories.class)
public class TheoryPassedTest {

	@Theory
	public void theories() {
		Assume.assumeTrue(true);
	}
}

[sbabcoc]

Yes, I didn't add support for "theories". JUnit Foundation currently only supports the @Test annotation. I'll see how much work will be required to add support for "theories".

[sbabcoc]

I'm adding support for the Theories runner. There's a significant issue with the core JUnit implementation of this runner, though - it doesn't report each individual iteration of the "theory" methods it runs. It reports each "theory" as a single test, regardless of how many applicable assumptions there are. Also, iteration terminates if a failure is encountered, even if there are data points that have yet to be evaluated.
JUnit Foundation reports the invocation of each iteration of the "theory" method separately to all MethodWatcher listeners, so this can be worked around to some extent. However, I'm not sure it's worth the effort.

[sbabcoc]

@HardNorth - Release 12.4.0 has been published.

[sbabcoc]

@HardNorth - I'm working on a PR that adds lifecycle events for Theory method permutations: #80
To make this work, I need to generate method Descriptions with unique IDs that incorporate the values of the invocation arguments. These are hashed and appended to the display name as hexadecimal. This enables event subscribers to get the class runner that invoked them, which retains the arguments. This also enables you to differentiate between Theories runner method events and TheoryAnchor method permutation events.

[sbabcoc]

@HardNorth - Pull request #80 has been merged, and new release 12.4.1 has been published. I think you'll be pleased.

[HardNorth]

@sbabcoc Sure, thank you very much!

[HardNorth]

@sbabcoc Verified. I’ve updated our JUnit agent with new JUnit Foundation. Theories look great! I’ve implemented several tests and haven’t met any issues.

[sbabcoc]

Excellent news! Thanks for the feedback! I was pretty pleased with the sleight-of-hand Statement shuffle I came up with to get this to produce the full complement of lifecycle events.