Cli tools

[keski]

This pull request is not yet ready for merge but should be discussed.

This pull request adds the following CLI wrappers:

• hefquin - runs a query via the engine
• hefquin-server - starts the embedded Web service
• hefquin-pgmat - materializes the RDF view of the PG
• hefquin-pg - runs a SPARQL query over a PG store

For hefquin, hefquin-server, and hefquin-pgmat the wrapper script provides usage instruction and and some argument validation. However, I realized when working on hefquin-pg that there is a great risk of introducing discrepancies between the script the the underlying Java CLI files. Thus I think we should focus on improving the usage instructions, argument validation, and error messages on the Java side, and then drop these parts on the script side.

With respect to the hefquin-service, there was a need to add command-line configuration support. Thus the use of a separate configuration file has been dropped and System.properties is now instead used to configure the servlet. This also means that there is no longer any need for the ConfigLoader.

For hefquin-pgmat and hefquin-pg, I was unable to make it work on my side without also introducing authorization (i.e. authentication). But I was running Neo4J in a docker container so I'm not sure if this was simply preconfigured for me. Anyways, I added parameters for username/password and passed these on to the connection if provided.

Usage
To use the scripts, the bin/ file needs to be added to the path, and HEFQUIN_HOME must be pointing to a directory containing the uber jars for hefquin-cli and hefquin-server. For example, from the project root run:

$ mkdir libs/
$ mvn clean compile
$ cp hefquin-cli/target/hefquin-cli-0.0.4-SNAPSHOT.jar ./libs
$ cp hefquin-service/target/hefquin-service-0.0.4-SNAPSHOT.jar ./libs
$ export HEFQUIN_HOME=$(pwd)
$ export PATH=$PATH:$(pwd)/bin

Q: How much focus we should put into making the Java CLI more user friendly in terms of parameter validation, usage instructions, and help texts at this point?

[keski]

Also, I have yet to write the corresponding bat/ scripts for Windows machines.

[hartig]

Thanks for this PR! Here are some comments for changes in the actual code. I will write a separate comment to respond to your questions and the text in the PR description.

[hartig]

[...] However, I realized when working on hefquin-pg that there is a great risk of introducing discrepancies between the script the the underlying Java CLI files. Thus I think we should focus on improving the usage instructions, argument validation, and error messages on the Java side, and then drop these parts on the script side.

Yes! The shell scripts should really just be simple wrappers around the Java programs and it makes no sense to duplicate usage instructions, argument validation, etc.

Is there any specific issue in the current Java-side usage instructions, argument validation, etc. that you didn't like and, thus, made you start replicating this functionality in the scripts?

For hefquin-pgmat and hefquin-pg, I was unable to make it work on my side without also introducing authorization (i.e. authentication). But I was running Neo4J in a docker container so I'm not sure if this was simply preconfigured for me.

Probably yes. I have so far only played around with a Neo4j instance that I start natively on my computer, and that one doesn't need a username/password.

Anyways, I added parameters for username/password and passed these on to the connection if provided.

Looks good. That would have been another TODO anyways. I think we may still want to make that a bit more robust in the future, but for the moment your extension is a good intermediate solution.

Usage
[...]

$ mkdir libs/
$ mvn clean compile
$ cp hefquin-cli/target/hefquin-cli-0.0.4-SNAPSHOT.jar ./libs
$ cp hefquin-service/target/hefquin-service-0.0.4-SNAPSHOT.jar ./libs

I don't like the need to create this extra directory (libs/) and then copy the JARs into it.

... at least not when working on the source code package. For the release packages that contain only the binaries, however, it makes sense to have all the relevant JARs in such a libs/ directory. Therefore, to be able to use the same shell scripts for both use cases, would it be possible to have the scripts check whether the required JAR is in the libs/ directory and use it in this case; and if it is not there, then check in the relevant target subdirectory (e.g., hefquin-cli/target/) ?

[hartig]

This is a great idea! I've now implemented this as part of bin/common.sh. If HEFQUIN_HOME is set, it will look for the executablehefquin-cli JAR file under HEFQUIN_HOME/libs or HEFQUIN_HOME/hefquin-cli/target/.

If HEFQUIN_HOME is not set, it will use the script location as the starting point. [...]

Great! I will try it to confirm that it works on my computer as well.

This needs to be called separately:

$ mvn package -pl hefquin-service -P build-war

or both steps on a single line:

$ mvn clean package && mvn package -pl hefquin-service -P build-war

Note that we are not packaging the entire package twice, the extra command only rebuilds the hefquin-service as an uber war.

Okay, so this doesn't affect me unless I really want to use the WAR. In other words, if I want to use the built-in servlet container to run the service, I still need to do only mvn clean package and then call the ./bin/hefquin-server script, right?

I feel that the PR is a bit complicated right now... I think the branch is just about ready to be merged, but I'm unsure about what steps remain.

I see that you are still pushing commits. Shall I take a look at the PR now or wait until you are done?

[keski]

Now this is ready to be reviewed. Had forgotten to update hefquin-docker.

[keski]

Okay, so this doesn't affect me unless I really want to use the WAR. In other words, if I want to use the built-in servlet container to run the service, I still need to do only mvn clean package and then call the ./bin/hefquin-server script, right?

Correct.

[keski]

Updates to Neo4jConnectionFactory according to review discussion were done in separate commits.

[hartig]

@keski I was checking the CLI tools on my computer and had to make a change: The scripts wouldn't run with the given she bang #!/bin/sh. I had to mark them explicitly as Bash script, using the she bang #!/bin/bash (see commit 94e8c9e)

Additionally, I extended common.sh with some error checking and error messages related to the JAR file to be discovered (see commit 0454f70).

Please check that things still work on your side.

[hartig]

Regarding the separate compilation of the WAR file (which you mention above), it doesn't work on my computer. More specifically, after successfully running

mvn clean package

I do

mvn package -pl hefquin-service -P build-war

which gives me the following error message.

[INFO] Scanning for projects...
[INFO] 
[INFO] -----------------< se.liu.ida.hefquin:hefquin-service >-----------------
[INFO] Building hefquin-service 0.0.4-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[WARNING] The POM for se.liu.ida.hefquin:hefquin-engine:jar:0.0.4-SNAPSHOT is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.584 s
[INFO] Finished at: 2024-09-25T14:27:19+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project hefquin-service: Could not resolve dependencies for project se.liu.ida.hefquin:hefquin-service:jar:0.0.4-SNAPSHOT: Could not find artifact se.liu.ida.hefquin:hefquin-engine:jar:0.0.4-SNAPSHOT -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

Do you have any idea how to fix this issue?

[keski]

Regarding the separate compilation of the WAR file (which you mention above), it doesn't work on my computer. More specifically, after successfully running

mvn clean package

I do

mvn package -pl hefquin-service -P build-war

which gives me the following error message.

[INFO] Scanning for projects...
[INFO] 
[INFO] -----------------< se.liu.ida.hefquin:hefquin-service >-----------------
[INFO] Building hefquin-service 0.0.4-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[WARNING] The POM for se.liu.ida.hefquin:hefquin-engine:jar:0.0.4-SNAPSHOT is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.584 s
[INFO] Finished at: 2024-09-25T14:27:19+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project hefquin-service: Could not resolve dependencies for project se.liu.ida.hefquin:hefquin-service:jar:0.0.4-SNAPSHOT: Could not find artifact se.liu.ida.hefquin:hefquin-engine:jar:0.0.4-SNAPSHOT -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

Do you have any idea how to fix this issue?

Could you try running mvn clean install before building the war?

[keski]

@keski I was checking the CLI tools on my computer and had to make a change: The scripts wouldn't run with the given she bang #!/bin/sh. I had to mark them explicitly as Bash script, using the she bang #!/bin/bash (see commit 94e8c9e)

Additionally, I extended common.sh with some error checking and error messages related to the JAR file to be discovered (see commit 0454f70).

Please check that things still work on your side.

Running the scripts with bash (#!/bin/bash) works fine on my end. I like the error checking, very user friendly!

[hartig]

Regarding the separate compilation of the WAR file (which #359 (comment)), it doesn't work on my computer.
[...]

Could you try running mvn clean install before building the war?

That works.

But doing it this way means that creating the WAR would require one to always deploy/install the JARs locally first. While that's not so nice, I can live with it. However, it also means that invoking both steps on a single line:

mvn clean package && mvn package -pl hefquin-service -P build-war

does not actually do what one would expect; i.e., the WAR is not built using the JARs created in the first step but, instead, using some JARs (of the same version) that have been deployed/installed earlier using mvn install. This might be confusing and lead to issues. So, we would really need to be extra careful when creating the artifacts for a (binary) release.

Do you still have an idea how to improve this part? Otherwise, we will have to live with it.

(The rest of the PR is okay now. Thanks!!!)

[keski]

I just found that there is an option that tells Maven to also build all required dependencies hefquin-service depends on (-am). Does the following work for you?

mvn package -pl hefquin-service -P build-war -am

[hartig]

[...] Does the following work for you?

mvn package -pl hefquin-service -P build-war -am

Perfect, that works! (yes, I removed the earlier-installed JARs from my ~/.m2/ directory before testing ;-)

So, the one-line version for compiling and packaging everything (the JARs and the WAR) then is:

mvn clean package && mvn package -pl hefquin-service -P build-war -am

I will merge the PR now :-)