Allow overriding prefix for native deps on a per-jar basis

[brandonbloom]

I'm trying to use LWJGL with Clojure, but am encountering some difficulty with the native libraries.

Two dependencies are required:

[org.lwjgl.lwjgl/lwjgl "2.8.5"]
[org.lwjgl.lwjgl/lwjgl-platform "2.8.5"]])

However, the lwjgl-platform dependency failed to resolve until I manually specified a classifier:

[org.lwjgl.lwjgl/lwjgl-platform "2.8.5" :classifier "natives-osx"]

That breaks the portability of my project.clj file, but that's probably a separate ticket.

The lwjgl-platform resources are available here on maven central

When I try to use LWJGL like this, I get this error:
UnsatisfiedLinkError no lwjgl in java.library.path

I did some digging and discovered that the native path is only added to your java.library.path if such a native platform directory exists. Upon further digging, it appears that no native libraries were being extracted because of this condition in extract-native-deps and the fact that the libraries in the jar are not nested in a native directory:

$ pwd
/Users/brandon/.m2/repository/org/lwjgl/lwjgl/lwjgl-platform/2.8.5
$ jar -tf lwjgl-platform-2.8.5-natives-osx.jar     
META-INF/
META-INF/MANIFEST.MF
liblwjgl.jnilib
openal.dylib

If I manually copy these files into target/native/macosx/x86_64, then LWJGL works as advertised!

I don't know how common it is have native libraries without a native/ directory, but I found at least one other example: jinput.

[technomancy]

To catch this up with the discussion on IRC: there is no standard JVM mechanism for storing native deps, so jars that aren't designed to work with Leiningen are unlikely to work out of the box. However, we could support a bit more flexibility by accepting a :native-prefix setting on a per-dependency basis that might allow this to work.

[marcliberatore]

I'd like this functionality too, so I thought I'd take a crack at this issue. I ran into difficulty.

It looks to me like per-dependency-vector settings are handled in pomegranate, not Leiningen. By the time we're in the scope of extract-native-deps, we're down to the sequence of jars that resolve-dependencies returns, which is fairly far removed from the per-dependency vector.

I'm not sure how best to proceed and would love some advice. Here are two ideas both of which seem to me to have flaws:

• Associate each vector's original form with each jar/zip it pulls in, and pass around this, rather than the sequence of jars alone. Then use that in extract-native-deps to determine what gets extracted. (Currently it's all pathnames within the jar/zip that start with native/.) But that seems very messy to me.
• Add a global project key that determines what should be extracted from natives. Or hardcode it. In my very short amount of poking around, Windows natives are .dlls, Linux natives are .sos, and Mac natives are .dylibs and/or .jnilibs. Ugly, and I assume this approach was rejected last time native support was updated.

Thoughts from those who understand the interplay of Leiningen and pomegranate would be most appreciated. Perhaps there's a function I'm overlooking that, say, given a jar returns the dependency vector(s) that pulled it in?

[technomancy]

We extract everything under native so that you can share the same checkout between OSes; for instance when hacking using vagrant. So the second bullet point isn't really an issue; it's not a per-os problem so much as deciding what prefix to use in the place of native.

I know we have been considering making pomegranate return more detailed information in order to support #642, so @cemerick might be up for adding some extra metadata to support this as well.

[cemerick]

Pomegranate is returning full dependency information right here, it just happens to be distilled into Files right away.

[marcliberatore]

Thanks. I had overlooked the metadata on the dependency tree's keys. I'll take a more careful look later, but I think that it should suffice for my purposes.

[marcliberatore]

I took a crack at addressing this issue in 4c98505. The idea is that you add a :native-prefix to any dependency vector where the default native/ is incorrect.

I went through several approaches before settling on this one. I modified the api of extract-native-deps slightly. @brandonbloom's example works for me now using:

[org.lwjgl.lwjgl/lwjgl-platform "2.8.5" :classifier "natives-osx" :native-prefix ""]

as the dependency vector, and adding

:jvm-opts ["-Djava.library.path=target/native"]

to the project (see #354). Also, I had to use Java 6 on OS X as Java 7 and lwjgl have some problems on OS X at present. The lwjgl-platform jar includes the natives in its top level, hence the empty :native-prefix.

Some comments/questions:

1. In this particular example, lwjgl-platform jar includes a META-INF/ directory in its top level, which is then extracted into target/native. I'm not sure this matters. Should it be special-cased not to be extracted?
2. I didn't test this patch extensively, and would love to know about other natives I can/should test against. Suggestions?
3. Is modifying the API for this issue unnecessary? I didn't see another way to do it.
4. Should I add an example to sample.project.clj?
5. I'm by no means a Clojure expert (or Leiningen expert or pomegranate expert...). I felt at times like I was doing some wheel-reinventing, despite my best efforts at code reuse. If I did reinvent some things, please let me know and I'll modify accordingly.

[technomancy]

Thanks for giving this a go. Do you mean that you have to add :jvm-opts to project.clj itself for this to work? I believe the current native-deps support works without adding anything, so it'd be nice if this could be added implicitly when native deps are detected.

Don't worry about META-INF/; extra files extracted shouldn't be a problem.

Adding an example to sample.project.clj would be great.

I have merged this, but it'd be great if we could get more testing from others who are using native dependencies.

Thanks!

[marcliberatore]

Yes, currently, for natives with non-Leiningen-standard layouts, this patch requires manually setting the java.library.path via :jvm-opts. I will look into making this automatic soonish. I will also update sample.project.clj.

[technomancy]

Hey, if this could make it in soon that would be great; hoping to cut a release at clojure/west next week. If not no worries; it can go into the next release. Thanks!

[marcliberatore]

The current code that sets the java.library.path property appears to clobber the default value for that property when natives that are designed to work with Leiningen are discovered (see here). The simplest thing to do, I think, is to continue this behavior, by modifying (native-arch-path) to look for and include any :native-prefix values in the dependencies. I'm going to take a crack at this shortly.

Longer term, I think it might be a good idea to consider merging the default value of java.library.path with the value entailed by :native-path and any :native-prefixes, but that's beyond the scope of this issue.

[marcliberatore]

In my very limited testing, 43581e2 works as intended. To quote technomancy, "it'd be great if we could get more testing from others who are using native dependencies."