db-shootout does not work on linux ppc64le / s390x

[MBaesken]

It seems the db-shootout benchmark does not work on linux ppc64le / s390x .
It contains a native lib (leveldbjni64-1.8 ) that is not available on all CPU/HW platforms where OpenJDK is supported.
Error message (example is from linux ppc64le)
java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, /tmp/libleveldbjni-64-1-8490766625802501468.8: /tmp/libleveldbjni-64-1-8490766625802501468.8: cannot open shared object file: No such file or directory (Possible cause: can't load AMD 64-bit .so on a Power PC 64 LE-bit platform)]

Probably the db-shootout benchmark should be run only on the platforms that are supposed to work (means: has the native library available).

[MBaesken]

btw. the reported issues has been seen with both JDK8 and JDK11.

[farquet]

Thanks Matthias for the report.
The leveldbjni-all dependency is "supposed" to support all platforms, unfortunately, it doesn't seem to be the case as one can see by unzipping the jar at http://central.maven.org/maven2/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar

I see different options here :

1. Add the missing native libs to a fork of leveldbjni-all and push it to maven since the owners don't seem willing to do it : New leveldbjni release to fix Hadoop yarn timeline server in PowerPC fusesource/leveldbjni#85
2. Retire the benchmark since it would be convenient not to have benchmarks relying on native libs for compatibility reasons. That would require a good replacement benchmark though.
3. Mark supported platforms for each benchmark and skip them when running on an unsupported platform.
4. Remove LevelDB from that benchmark since the benchmark consists of four different DBs tested. That probably wouldn't hurt so much to go down to three. https://github.com/renaissance-benchmarks/renaissance/blob/master/benchmarks/database/src/main/scala/org/renaissance/database/DbShootout.scala#L69-L91

Any opinion welcome @lbulej, @vhotspur, @axel22 & co.

I personally think that option 4 may be a good tradeoff. It's a simple change that would solve that particular issue. Moreover, the benchmark would probably not behave very differently. That would need to be quantify though.

[axel22]

I also think that option 4 is preferable for now.

[lbulej]

I also think that option 4 is preferable for now.

At the moment, adding support for 3 would be quite far in the priority chain of design changes, so I'd support 4 as well.

[axel22]

Wouldn't the list of supported platforms for each benchmark just be a property associated with a benchmark?

[farquet]

Yes, that would be just a property, but if we can slightly change the problematic benchmark(s) to work on most platforms, that’s one less dimension to care about.

I will tackle the implementation of 4 by measuring the performance difference on several VMs.

[farquet]

@MBaesken, the benchmark has been fixed to remove the native dependency.
Please give it a try and reopen the issue if there are remaining problems.

You can build a jar from master, or wait for the 0.10.0 release which will come very soon.

[MBaesken]

When testing the latest renaissance-mit-0.10.0.jar db-shootout I still run into issues with native libs that are not found .

AIX :

/rs6000_64/nightly/output-jdk8/images/j2sdk-image/bin/java -jar /benchmarks/renaissance/renaissance-mit-0.10.0.jar db-shootout
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Error during tear-down: null
java.lang.NullPointerException
at org.lmdbjava.bench.Chronicle$CommonChronicleMap.teardown(Chronicle.java:131)
at org.lmdbjava.bench.Chronicle$Reader.teardown(Chronicle.java:210)
at org.renaissance.database.DbShootout.tearDownAfterAll(DbShootout.scala:85)
at org.renaissance.RenaissanceBenchmark.runBenchmark(RenaissanceBenchmark.java:102)
at org.renaissance.RenaissanceSuite$.$anonfun$main$2(renaissance-suite.scala:298)
at org.renaissance.RenaissanceSuite$.$anonfun$main$2$adapted(renaissance-suite.scala:296)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.renaissance.RenaissanceSuite$.main(renaissance-suite.scala:296)
at org.renaissance.RenaissanceSuite.main(renaissance-suite.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.renaissance.Launcher.main(Launcher.java:21)
Exception occurred in org.renaissance.database.DbShootout@2f666ebb: Native library (com/sun/jna/aix-ppc64/libjnidispatch.so) not found in resource path ...
.......
at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:866)
at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826)
at com.sun.jna.Native.(Native.java:140)
at com.sun.jna.NativeLibrary.(NativeLibrary.java:84)
at net.openhft.chronicle.hash.impl.util.jna.PosixMsync.(PosixMsync.java:28)
at net.openhft.chronicle.hash.impl.VanillaChronicleHash.msync(VanillaChronicleHash.java:845)
at net.openhft.chronicle.hash.impl.VanillaChronicleHash.msync(VanillaChronicleHash.java:831)
at net.openhft.chronicle.map.ChronicleMapBuilder.commitChronicleMapReady(ChronicleMapBuilder.java:413)
at net.openhft.chronicle.map.ChronicleMapBuilder.createWithNewFile(ChronicleMapBuilder.java:1749)
at net.openhft.chronicle.map.ChronicleMapBuilder.createWithFile(ChronicleMapBuilder.java:1652)
at net.openhft.chronicle.map.ChronicleMapBuilder.createPersistedTo(ChronicleMapBuilder.java:1552)
at org.lmdbjava.bench.Chronicle$CommonChronicleMap.setup(Chronicle.java:122)
at org.lmdbjava.bench.Chronicle$Reader.setup(Chronicle.java:204)
at org.renaissance.database.DbShootout.setUpBeforeAll(DbShootout.scala:73)
at org.renaissance.RenaissanceBenchmark.runBenchmark(RenaissanceBenchmark.java:79)
at org.renaissance.RenaissanceSuite$.$anonfun$main$2(renaissance-suite.scala:298)
at org.renaissance.RenaissanceSuite$.$anonfun$main$2$adapted(renaissance-suite.scala:296)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.renaissance.RenaissanceSuite$.main(renaissance-suite.scala:296)
at org.renaissance.RenaissanceSuite.main(renaissance-suite.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.renaissance.Launcher.main(Launcher.java:21)

linuxs390x (OpenJDK jdk11, I had no jdk8 at hand) :

/linuxs390x/nightly/output-jdk11-test/images/jdk/bin/java -jar /benchmarks/renaissance/renaissance-mit-0.10.0.jar db-shootout
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Error during tear-down: null
java.lang.NullPointerException
at org.lmdbjava.bench.Chronicle$CommonChronicleMap.teardown(Chronicle.java:131)
at org.lmdbjava.bench.Chronicle$Reader.teardown(Chronicle.java:210)
at org.renaissance.database.DbShootout.tearDownAfterAll(DbShootout.scala:85)
at org.renaissance.RenaissanceBenchmark.runBenchmark(RenaissanceBenchmark.java:102)
at org.renaissance.RenaissanceSuite$.$anonfun$main$2(renaissance-suite.scala:298)
at org.renaissance.RenaissanceSuite$.$anonfun$main$2$adapted(renaissance-suite.scala:296)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.renaissance.RenaissanceSuite$.main(renaissance-suite.scala:296)
at org.renaissance.RenaissanceSuite.main(renaissance-suite.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.renaissance.Launcher.main(Launcher.java:21)
Exception occurred in org.renaissance.database.DbShootout@29a60c27: Native library (com/sun/jna/linux-s390x/libjnidispatch.so) not found
......
at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:866)
at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826)
at com.sun.jna.Native.(Native.java:140)
at net.openhft.chronicle.hash.impl.util.jna.PosixFallocate.(PosixFallocate.java:18)
at net.openhft.chronicle.hash.impl.VanillaChronicleHash.map(VanillaChronicleHash.java:977)
at net.openhft.chronicle.hash.impl.VanillaChronicleHash.createMappedStoreAndSegments(VanillaChronicleHash.java:485)
at net.openhft.chronicle.map.ChronicleMapBuilder.createWithNewFile(ChronicleMapBuilder.java:1748)
at net.openhft.chronicle.map.ChronicleMapBuilder.createWithFile(ChronicleMapBuilder.java:1652)
at net.openhft.chronicle.map.ChronicleMapBuilder.createPersistedTo(ChronicleMapBuilder.java:1552)
at org.lmdbjava.bench.Chronicle$CommonChronicleMap.setup(Chronicle.java:122)
at org.lmdbjava.bench.Chronicle$Reader.setup(Chronicle.java:204)
at org.renaissance.database.DbShootout.setUpBeforeAll(DbShootout.scala:73)
at org.renaissance.RenaissanceBenchmark.runBenchmark(RenaissanceBenchmark.java:79)
at org.renaissance.RenaissanceSuite$.$anonfun$main$2(renaissance-suite.scala:298)
at org.renaissance.RenaissanceSuite$.$anonfun$main$2$adapted(renaissance-suite.scala:296)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.renaissance.RenaissanceSuite$.main(renaissance-suite.scala:296)
at org.renaissance.RenaissanceSuite.main(renaissance-suite.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.renaissance.Launcher.main(Launcher.java:21)

So it looks like there are native dependencies remaining.

[axel22]

I think we should maybe think about including the Apache Ignite database in this benchmark instead.

[farquet]

Thanks @MBaesken for the testing. This is where the native library is loaded in Chronicle code : https://github.com/OpenHFT/Chronicle-Map/blob/master/src/main/java/net/openhft/chronicle/hash/impl/util/jna/PosixMsync.java#L28

I'll investigate how much it take to replace Chronicle by Ignite.

[lbulej]

Thanks @MBaesken for the testing. This is where the native library is loaded in Chronicle code : https://github.com/OpenHFT/Chronicle-Map/blob/master/src/main/java/net/openhft/chronicle/hash/impl/util/jna/PosixMsync.java#L28

I'll investigate how much it take to replace Chronicle by Ignite.

Or we can just update the JNA library used by Chronicle to call msync. JNA 5.9.0 seems to have support for ppc64le/s390x, so it might help. I did the update in an attempt to fix one of the issues in #296 (which is common to this one) so it might be relevant to this issue as well. See my comment in #296 for link to a custom build of Renaissance containing only db-shootout with the updated JNA library.

[MBaesken]

Linux ppc64le worked nicely when testing with renaissance-gpl-0.13.0-4-g2755c79.jar (both with our JDK8 and JDK11).
AIX worked too when testing with JDK8 (had no JDK11 at hand).
Linux s390x worked with JDK11; JDK8 failed with the exception

Benchmark 'db-shootout' failed with exception:
java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/linux-zarch_64/libjnidispatch.so) not found in resource path

While os.arch is s390x in JDK11, it was still zArch_64 in 8 . This might lead to the error.

[lbulej]

@MBaesken That's excellent news! It seems that mapping from zArch_64 to s390x should be easy enough to handle in JNA, so I guess we can delay merging #322 until the fix makes it to JNA. Are you going to open an issue with the JNA people?

[MBaesken]

I opened java-native-access/jna#1388 at JNA regarding the os.arch mapping; let's see what happens.

[farquet]

Can this be closed now?

[MBaesken]

Hi, my change for issue java-native-access/jna#1388 went into JNA .
There is already a JNA 5.10 containing it (Map the arch zarch_64 as reported by SAPJVM8) : https://github.com/java-native-access/jna/blob/master/CHANGES.md#release-5100
did you switch to this new version already , if so then the issue can be closed.

[farquet]

@lbulej's PR bumps to the new version: #322

[MBaesken]

Sounds great - any idea when the new release with this change approx. will be released ?
(and yes I think we can close this issue)

[lbulej]

Sounds great - any idea when the new release with this change approx. will be released ? (and yes I think we can close this issue)

The plan was to get a maintenance update out by the beginning of December but we wanted to make it easier to run the benchmarks on platforms that cannot do multiple class loaders. If things go well now, the first half of January 2022 looks like a reasonable target.

[lbulej]

It took a little longer, but the updated JNA that fixes the problem is now part of the 0.14.0 release. Thank you for helping with this issue!