ImportError: librdkafka.so.1: cannot open shared object file: No such file or directory

[pwaterz]

Currently running into this error

$ python2.7
Python 2.7.12 (default, Jun 28 2016, 17:49:25)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from confluent_kafka import Consumer, KafkaError
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/site-packages/confluent_kafka/__init__.py", line 2, in <module>
    from .cimpl import (Consumer,
ImportError: librdkafka.so.1: cannot open shared object file: No such file or directory

When trying to import the package.
I'm running this on REHL 5 and installed the package via

pip2.7 install confluent-kafka

Any ideas?

[edenhill]

Hi Patrick W (@pwaterz) ,

did you install librdkafka first?
If so, how? (yum, build from source, ..?)

[pwaterz]

I did, I built it from source

 ls -la /usr/local/include/librdkafka/
total 200
drwxr-xr-x 2 root root  4096 Mar 21 10:38 .
drwxr-xr-x 8 root root  4096 Mar  9 12:02 ..
-rwxr-xr-x 1 root root 65663 Mar 21 10:38 rdkafkacpp.h
-rwxr-xr-x 1 root root 99635 Mar 21 10:38 rdkafka.h

[edenhill]

It could be that the linker doesnt look in /usr/local/lib, you can verify this by doing:
LD_LIBRARY_PATH=/usr/local/lib python -m confluent_kafka

If that works it means you should add /usr/local/lib to /etc/ld.so.conf and rerun ldd (iirc).

Another alternative is to install librdkafka in the system location right away, by:

cd librdkafka
./configure --prefix=/usr
make clean all
sudo make install

[pwaterz]

Doesn't look like that's the issue

$ LD_LIBRARY_PATH=/usr/local/lib python2.7 -m confluent_kafka
/usr/bin/python2.7: No module named confluent_kafka.__main__; 'confluent_kafka' is a package and cannot be directly executed

The kafka lib is installed here

/usr/lib64/python2.7/site-packages/confluent_kafka/

[edenhill]

Sorry, try this:
LD_LIBRARY_PATH=/usr/local/lib python2.7 -m confluent_kafka.kafkatest.verifiable_client

Or just start python with LD_LIBRARY_PATH=... and do the manual 'import confluent_kafka'

[pwaterz]

That did not error. I will try and fix the linking in the library and get back to you. Thanks a lot for your time!

[edenhill]

Dup of #65

[dogyy0119]

I just ldconfig
after make install , then it works

[vikramj74]

Thanks lliuhang (@dogyy0119) , ldconfig worked for me.

[idati]

I just ldconfig
after make install , then it works

Magnus Edenhill (@edenhill) please can you add it to the README, that would be great :-)