What is the correct way to test in "module aware" mode?

[io7m]

Hello!

I have rather a lot of projects that use the Java module system. I don't care about testing artifacts in classpath mode as I only support using the artifacts I produce on the module path.

Here's a project I'll use for the sake of this discussion, but I've experienced variations of the same issue on countless other projects:

https://github.com/io7m/cedarbridge

Edit: The project above doesn't have any of the workarounds I mention below; it's a blank slate right now

In order to make JUnit and/or Intellij IDEA's test runner and/or Maven Surefire run the test code such that the module path is used rather than the classpath, I have to add a module descriptor. Because I often have projects that have dozens of individual modules, I have all of the tests in one module. Here's the cedarbridge example:

https://github.com/io7m/cedarbridge/tree/develop/com.io7m.cedarbridge.tests

If I add a module descriptor in src/test/java in the com.io7m.cedarbridge.tests module, the Maven compiler plugin complains:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile (default-testCompile) on project com.io7m.cedarbridge.tests: Execution default-testCompile of goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile failed: Can't compile test sources when main sources are missing a module descriptor -> [Help 1]
[ERROR]

I can instead add a module descriptor in src/main/java in the com.io7m.cedarbridge.tests module.
Unfortunately, this then causes Surefire to break when the tests run:

[ERROR] class org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder (in unnamed module @0x33723e30) cannot access class org.junit.platform.commons.util.Preconditions (in module org.junit.platform.commons) because module org.junit.platform.commons does not export org.junit.platform.commons.util to unnamed module @0x33723e30

As well as trying to run the tests in the IDE:

Exception in thread "main" java.lang.IllegalAccessError: class org.junit.platform.engine.UniqueId (in unnamed module @0x6acdbdf5) cannot access class org.junit.platform.commons.util.Preconditions (in module org.junit.platform.commons) because module org.junit.platform.commons does not export org.junit.platform.commons.util to unnamed module @0x6acdbdf5
	at org.junit.platform.engine.UniqueId.forEngine(UniqueId.java:69)
	at com.intellij.junit5.JUnit5IdeaTestRunner.<clinit>(JUnit5IdeaTestRunner.java:72)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:391)
	at java.base/java.lang.Class.forName(Class.java:382)
	at com.intellij.rt.junit.JUnitStarter.getAgentClass(JUnitStarter.java:241)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:222)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)

I've seen lots of variations on this error on different projects, but the org.junit.platform.commons frequently seems to be mentioned.

If I add identical module descriptors at both src/main/java/module-info.java and src/test/java/module-info.java, this kind of works in that Maven's compiler plugin accepts it, but it does mean maintaining two module descriptors, and it also means that for each real test package in src/test/java, I have to have a mirror fake package in src/main/java with an empty class in each to allow javac to accept the module descriptor.

Assuming I actually do all of the above, I can get tests to run if I add the following VM options:

--add-opens org.junit.platform.commons/org.junit.platform.commons.logging=ALL-UNNAMED --add-opens org.junit.platform.commons/org.junit.platform.commons.util=ALL-UNNAMED

... But this is fairly miserable, as if I click a test to run it in the IDE, it will obviously fail immediately with an IllegalAccessError, and I then have to go and add those VM options to the test configuration and run it again.

I want to run tests in module-path mode only, and I'd like to know if there's some strategy I can use across ~120 projects that all have largely the same structure without needing extra VM options, multiple descriptors, and all that...

I'm opening the discussion here because I don't know which software is at fault.

[sormuras]

Hi Mark, author of at least 182 unique Java modules published to Maven Central.

In the blog "Testing in The Modular World" (2018) I wrote about and created a sample project covering two ways of in-module testing:

• Patching test binaries into main modules at runtime
• Patching main sources into test modules at compile time

As IDEs and Maven (that is the Compiler Plugin and Surefire) didn't support those features back then (at all, nor do they agree today), I also created the "JUnit Platform Maven Plugin". This plugin might mend the cases in your projects in which Surefire breaks by not running on the module path itselfs.

[...] I don't care about testing artifacts in classpath mode as I only support using the artifacts I produce on the module path.
[...]
I want to run tests in module-path mode only, and I'd like to know if there's some strategy I can use across ~120 projects that all have largely the same structure without needing extra VM options, multiple descriptors, and all that...

Tests, in general, should be written as close as possible to real code your users would do in their main code. Your tests are your first customers, so to say. From your goal above ("only support using the artifacts I produce on the module path") you are doing right by wanting "to run tests in module-path mode only". That means, customers (your users AND your tests) of your (main) modules must configure the module system correctly. There is no way to work around "all that"; not for your users NOR your tests.

You either need to supply and maintain extra module descriptors for testing (as your users do for their modules) or use the super-powers of the command line via extra VM options.

@michael-simons wrote about that very topic here: https://info.michael-simons.eu/2021/10/19/testing-in-a-modular-world/

[io7m]

Hello!

Hi Mark, author of at least 182 unique Java modules published to Maven Central.

😅

I think it might actually be significantly more than that by now. For whatever reason, that list is missing quite a few things.

I also created the "JUnit Platform Maven Plugin". This plugin might mend the cases in your projects in which Surefire breaks by not running on the module path itselfs.

Thanks, I'll take a look at this. That's not going to help tests not running properly in the IDE, though, right?

Tests, in general, should be written as close as possible to real code your users would do in their main code.

I feel like I'm doing exactly that. The tests are in a different source root, but that would be true for any users writing code using the artifacts too.

That means, customers (your users AND your tests) of your (main) modules must configure the module system correctly. There is no way to work around "all that"; not for your users NOR your tests.
You either need to supply and maintain extra module descriptors for testing (as your users do for their modules) or use the super-powers of the command line via extra VM options.

This is where I feel like something is broken somewhere. The main artifacts in all of my projects absolutely do have correct module descriptors (I know because I use lots of my own code in modular projects). The test code only runs against code that is exported in module descriptors; sometimes I'll exports com.io7m.xyz.internal to com.io7m.xyz.tests if I need to specifically test non-exported packages.

I'm having to maintain extra module descriptors and use extra VM options despite not doing any kind of magic with my code, or deliberately breaking into the internals of modules. That seems wrong. If we assume that programming with modules is ostensibly the new "standard" way of writing Java code, then I feel like I shouldn't be having to do both of the things above (one correct module descriptor should be enough). Unfortunately, I don't know which of five different systems is at fault (javac, Surefire, Maven compiler plugin, Intellij IDEA, JUnit...).

I'd like to try to get this working properly.

wrote about that very topic here: https://info.michael-simons.eu/2021/10/19/testing-in-a-modular-world/

Reading this now.

[io7m]

Hm, actually, that's one thing I haven't tried: Put all of the tests in the test module in src/main/java and tell Surefire to find tests there instead. I'll try that today and see if it improves matters.

[io7m]

I'll also try opens/exports com.io7m.xyz.tests to org.junit.platform.commons;.

[sormuras]

This is where I feel like something is broken somewhere. The main artifacts in all of my projects absolutely do have correct module descriptors (I know because I use lots of my own code in modular projects). The test code only runs against code that is exported in module descriptors; sometimes I'll exports com.io7m.xyz.internal to com.io7m.xyz.tests if I need to specifically test non-exported packages.

What you describe here, should be the easy and most realistic way of running tests in the modular world.

You need to declare an internal module descriptor for testing though, that could follow this structure:

open /* to allow deep reflection for test frameworks */ module com.io7m.xyz.tests {
  //
  // modules under test
  //
  requires ...;
  requires ...;

  //
  // test framework api
  //
  requires org.junit.jupiter.api;
  requires net.jqwik.api;
}

IDEs and build tools should understand that "as-is", at least they did, when I last checked.

[io7m]

I don't want to speak too soon, but it seems like these were the key things that worked:

open module com.io7m.xyz.tests
^^^^ // Missing this, or not opening the module to org.junit.platform.commons would cause errors.
{
  requires ...;

  // Missing any one of these four caused a variety of different errors.
  requires org.junit.jupiter.api;
  requires org.junit.jupiter.engine;
  requires org.junit.platform.commons;
  requires org.junit.platform.engine;

  exports com.io7m.ervilla.tests;
}

This module descriptor has to be duplicated and one placed in src/main/java and one in src/test/java. That's not great, but it's manageable. With this setup, Surefire runs the tests in module path mode, and the IDE also runs them in module path mode without either of them needing any extra VM options.

I'm going to experiment with using src/main/java for all tests given that I don't have tests in any modules other than the tests module. That might let me eliminate the duplicated descriptors.

[io7m]

I've landed at moving all tests into src/main/java and configuring Surefire and Failsafe to find tests there.

This works in that Surefire and Failsafe run the tests in module-path mode without needing any VM arguments.

It does, however, fail in Intellij IDEA without extra VM arguments. I'm pretty sure the way they're running JUnit is just wrong; it's always on the classpath even if everything else is on the module path.

Here's a project with this setup: https://github.com/io7m/idstore