Multi-module liberty dev mode fails when there exists a test-jar dependency

[lmsurpre]

I have an existing multi-module web application and I want to use liberty dev mode with it.

One of the modules has some test utilities which are used in the tests of the other modules.
Maven supports that through the use of a build plugin configuration like this:

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>3.2.2</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>test-jar</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

Then, in the modules that depend on it, we declare the dependency with a type of test-jar (or, equivalently, with a classifier of tests)

This builds fine with a command like mvn clean install -f parent. However, if one of those modules has our liberty config and we try using mvn liberty:dev -f parent then we end up with an error like the following:

Could not resolve dependencies for project io.openliberty.guides:module2:war:1.0-SNAPSHOT: Could not find artifact io.openliberty.guides:module1:jar:tests:1.0-SNAPSHOT

Attached is a minimal reproducer that adapts the OpenLiberty getting started example to demonstrate the issue.
multimodule-testjar-reproducer.zip
After unzipping, run the following to see the error:

mvn liberty:dev -f parent

[lmsurpre]

I'll also note that the documented mvn liberty:dev -f parent -pl module2 -am syntax didn't work for me at all.
That one produces an error like this:

[ERROR] [ERROR] Could not find the selected project in the reactor: module2 @ 
[ERROR] Could not find the selected project in the reactor: module2 -> [Help 1]

However, using the "reactor" syntax (prefixing the module name with a : to indicate that module2 is an artifactId), I got back to the same issue described above).

I opened https://github.com/OpenLiberty/ci.maven/pull/1562/files to update the docs accordingly, but let me know if you'd like me to open a separate issue for that.

$ mvn liberty:dev -f parent -pl :module2 -am
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] module1                                                            [jar]
[INFO] module2                                                            [war]
[INFO] 
[INFO] -------------------< io.openliberty.guides:module1 >--------------------
[INFO] Building module1 1.0-SNAPSHOT                                      [1/2]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- liberty-maven-plugin:3.5.1:dev (default-cli) @ module1 ---
[INFO] CWWKM2102I: Using artifact based assembly archive : io.openliberty:openliberty-kernel:null:22.0.0.7:zip.
[INFO] CWWKM2102I: Using installDirectory : /Users/lsurprenant/git/guide-getting-started/module1/target/liberty/wlp.
[INFO] CWWKM2102I: Using serverName : defaultServer.
[INFO] CWWKM2102I: Using serverDirectory : /Users/lsurprenant/git/guide-getting-started/module1/target/liberty/wlp/usr/servers/defaultServer.
[INFO] Running maven-dependency-plugin:purge-local-repository
[INFO] Running maven-resources-plugin:resources
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/lsurprenant/git/guide-getting-started/module1/src/main/resources
[INFO] Running maven-compiler-plugin:compile on /Users/lsurprenant/git/guide-getting-started/module1/pom.xml
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] -------------------< io.openliberty.guides:module2 >--------------------
[INFO] Building module2 1.0-SNAPSHOT                                      [2/2]
[INFO] --------------------------------[ war ]---------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for module1 1.0-SNAPSHOT:
[INFO] 
[INFO] module1 ............................................ SUCCESS [  0.553 s]
[INFO] module2 ............................................ FAILURE [  0.004 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.637 s
[INFO] Finished at: 2022-07-12T15:37:30-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project module2: Could not resolve dependencies for project io.openliberty.guides:module2:war:1.0-SNAPSHOT: Could not find artifact io.openliberty.guides:module1:jar:tests:1.0-SNAPSHOT -> [Help 1]

[kathrynkodama]

Thanks @lmsurpre, I am able to reproduce this issue.

Looks as though dev mode is unable to resolve the module1 dependency in this case because of the tests classifier:

        <dependency>
            <groupId>io.openliberty.guides</groupId>
            <artifactId>module1</artifactId>
            <version>1.0-SNAPSHOT</version>
            <classifier>tests</classifier>
            <scope>test</scope>
        </dependency>

In dev mode we should support classifiers when resolving dependencies for multi module projects.

I'll also note that the documented mvn liberty:dev -f parent -pl module2 -am syntax didn't work for me at all. That one produces an error like this:

[ERROR] [ERROR] Could not find the selected project in the reactor: module2 @ 
[ERROR] Could not find the selected project in the reactor: module2 -> [Help 1]

However, using the "reactor" syntax (prefixing the module name with a : to indicate that module2 is an artifactId), I got back to the same issue described above).

I opened https://github.com/OpenLiberty/ci.maven/pull/1562/files to update the docs accordingly, but let me know if you'd like me to open a separate issue for that.

The documentation for the -pl module2 -am syntax assumes a relative path approach, for this project layout, -pl ../module2 -am would work. However, I think to be more robust documenting to always prefix the module name with a : to indicate artifactId is a good idea rather than the relative path approach.

More details on --projects (-pl for short) here: https://maven.apache.org/guides/mini/guide-multiple-modules-4.html#inclusion-and-exclusion

[maroschutte]

I just encountered this problem as well today. To me it seems related to 'maven-dependency-plugin:purge-local-repository' being called during execution of 'liberty-maven-plugin:dev' and I am wondering why this is being done. Does anyone have a pointer for me ?

[kathrynkodama]

I just encountered this problem as well today. To me it seems related to 'maven-dependency-plugin:purge-local-repository' being called during execution of 'liberty-maven-plugin:dev' and I am wondering why this is being done. Does anyone have a pointer for me ?

Hi @maroschutte, hot deployment through dev mode is made possible because the application is installed as a loose application. A loose application loads the associated files directly from the workspace rather than being exported.

Dev mode calls maven-dependency-plugin:purse-local-repository in a multi module scenario to ensure that any downstream modules will not rely on the installed artifacts for their compilation, but are instead loaded from the workspace (loose application).

Unfortunately we do not yet have a workaround for this sort of project configuration.

More on loose applications in Liberty here: https://www.ibm.com/docs/en/was-liberty/base?topic=liberty-loose-applications

[maroschutte]

It really worries me that liberty-maven-plugin clears artifacts from the local repository. At the very least it is really confusing to have artifacts that you put there using a 'mvn install' just disappear, but it could actually mean that existing setups that rely on artifacts being present in the localrepo break once you run dev mode. The test jars are a good example of this.

[scottkurz]

It really worries me that liberty-maven-plugin clears artifacts from the local repository. At the very least it is really confusing to have artifacts that you put there using a 'mvn install' just disappear, but it could actually mean that existing setups that rely on artifacts being present in the localrepo break once you run dev mode. The test jars are a good example of this.

That's a good question @maroschutte. I'm looking over the logic behind the change we made in #1201 and trying to understand better. If the purpose is just to surface a warning message to the user then messing with someone's local .m2 repo seems a bit disruptive. Will let you know what we find and create a new issue if necessary (since this one has already identified a primary issue with the use of the classifier). Thank you for bringing this up.

[scottkurz]

I opened a new issue #1569 to consider an alternative to doing the purge since I have a prototype which may solve the problem in a better way.

[lmsurpre]

I saw #1569 is marked as completed now... are we expecting that to also address the issue presented here?

[cherylking]

@lmsurpre Scott is on vacation right now, but I believe the changes for #1569 were meant to solve this. I'll wait for him to return and confirm before closing this issue though.

[scottkurz]

@lmsurpre good question. The #1569/#1570 change will get past the error in the recreate use case you mentioned, where you did the mvn install but then the dependency couldn't be resolved.

So from that perspective, the problem will be fixed in the next release.

You could also achieve something similar by doing: mvn clean test-compile jar:test-jar liberty:dev without needing to do the local install.

However, I wonder what your intention was in declaring the test dependency and I wonder if you are going to be happy with the dev mode behavior here. I can forsee a potential problem: if the idea was to provide failsafe configuration in module2 with a test packaged in module1 then I think this could be a gotcha, since I believe dev mode would likely run the module1 tests using the module1 pom config.

Perhaps I'm getting ahead of myself though. In any case, when the next release is available, please let us know if you have other issues, either by an issue on this repo or on Gitter at: https://gitter.im/OpenLiberty/developer-experience. Thanks.

[lmsurpre]

if the idea was to provide failsafe configuration in module2 with a test packaged in module1 then I think this could be a gotcha, since I believe dev mode would likely run the module1 tests using the module1 pom config.

I'd actually like to learn how to set up a test like that :-)
However, in the meantime, its more like we have tests in module2 that refer to test utilities in module1.
Happy to try this and report back whenever it gets released...is there any rough timeframe on that (e.g. days vs weeks vs months?)