pyarrow.lib.ArrowIOError: Unable to load libjvm

[jurgispods]

Hi,

I tried installing pyarrow under Cloudera CDH 5.12.1 and running this minimal script:

python
>>> import pyarrow as pa
>>> fs = pa.hdfs.connect("localhost", 50070, user="me")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/site-packages/pyarrow/hdfs.py", line 170, in connect
    kerb_ticket=kerb_ticket, driver=driver)
  File "/usr/lib64/python2.7/site-packages/pyarrow/hdfs.py", line 37, in __init__
    self._connect(host, port, user, kerb_ticket, driver)
  File "pyarrow/io-hdfs.pxi", line 79, in pyarrow.lib.HadoopFileSystem._connect (/arrow/python/build/temp.linux-x86_64-2.7/lib.cxx:61546)
  File "pyarrow/error.pxi", line 79, in pyarrow.lib.check_status (/arrow/python/build/temp.linux-x86_64-2.7/lib.cxx:8345)
pyarrow.lib.ArrowIOError: Unable to load libjvm

Some env info:

• HADOOP_HOME and JAVA_HOME are correctly set
• $JAVA_HOME/lib/amd64/server contains the file libjvm.so

Any help is appreciated!

[wesm]

see https://github.com/apache/arrow/blob/master/cpp/src/arrow/io/hdfs-internal.cc#L178

so it's $JAVA_HOME/lib/amd64/server and not $JAVA_HOME/jre/lib/amd64/server?

[jurgispods]

Here are the complete paths:

• JAVA_HOME:

$ echo $JAVA_HOME
/usr/lib/jvm/jre-1.8.0

• Path containing libjvm.so:

$ ls /usr/lib/jvm/jre-1.8.0/lib/amd64/server/
classes.jsa  libjsig.so  libjvm.so  Xusage.txt

[wesm]

Thanks! I opened https://issues.apache.org/jira/browse/ARROW-1966 about making the libjvm loading more lenient (not expecting JAVA_HOME to contain a jre directory like it used to in prior installs of the JDK... thanks Oracle)

[wesm]

Patches would be welcome for this

[wesm]

A patch is open #1487

[ucrai]

Please could you share with me:
how is the script called, where I can change the search path to libjvm? I cannot find a hdfs-internal.cc in my pyarrow distribution. (using pyarrow 0.8.0).
Thanks a whole lot.

[bbagherian]

I'm getting the same error in Windows, my JAVA_Home env is 'c:\java\bin' .... any idea?

[eromoe]

There is no such file.
Could you add a message to tell people how to deal this problem when raise Exception ?

[wesm]

Please open a JIRA issue if you are having trouble and describe your environment and how to reproduce the problem, as well as providing a complete traceback