Missing libpython with Debian default Python install

[matthew-brett]

I am testing manylinux numpy wheels.

In particular, I am testing this guy: http://nipy.bic.berkeley.edu/manylinux/numpy-1.10.4-cp27-none-linux_x86_64.whl

With a default install, test of Python, starting with either Wheezy or Jessie:

docker run -ti --rm -v $PWD:/io debian:latest /bin/bash
docker run -ti --rm -v $PWD:/io tianon/debian:wheezy /bin/bash

and running this script:

apt-get update 
apt-get install -y python curl 
curl -sLO https://bootstrap.pypa.io/get-pip.py 
python get-pip.py 
pip install -f https://nipy.bic.berkeley.edu/manylinux numpy nose 
python -c "import numpy; numpy.test()" 

I get this:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/numpy/__init__.py", line 180, in <module>
    from . import add_newdocs
  File "/usr/local/lib/python2.7/dist-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/usr/local/lib/python2.7/dist-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/usr/local/lib/python2.7/dist-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/usr/local/lib/python2.7/dist-packages/numpy/core/__init__.py", line 14, in <module>
    from . import multiarray
ImportError: libpython2.7.so.1.0: cannot open shared object file: No such file or directory

Sure enough, in order for the wheel to work, I need:

apt-get install libpython2.7

Maybe we need to add a check / informative error message for the presence of libpython?

[matthew-brett]

Same on Ubuntus 14.04, 15.10.

[rmcgibbo]

Where would such a check go? This is really a Debian problem with the way they split upstream python into many subpackages, no?

[matthew-brett]

It would have to go in the pip check for the validity of manylinux for the platform.

It is unfortunate that Debian does this, but I bet that a large majority of our consumers will be on Debian or Ubuntu, and there's not much prospect of Debian / Ubuntu changing this immediately, and even if they do, there will be many systems liable to this problem in the wild.

[rmcgibbo]

But on that system, if you disable manylinux wheels, pip install is going to pull from an sdist, and that will fail too.

[rmcgibbo]

Because clearly the system doesn't have the python header files (apt-get install python-dev), otherwise they'd also have libpython.

[matthew-brett]

Right - good point.

On the other hand, on a Debian-based system, if I try an sdist pip install I first get an error about being unable to compile a C program, and, after installing build-essential I get this:

    SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.

That makes it pretty clear what I have to do. On the other hand, if we blanket enable manylinux wheels, on systems without python-dev, the error (libpython2.7.so.1.0: cannot open shared object file) is much harder to understand and fix.

[rmcgibbo]

Do you know where that SystemError is being thrown from? Is that check in pypa/pip, or something numpy-specific.

[rmcgibbo]

I guess that's a numpy thing: https://github.com/numpy/numpy/blob/master/numpy/core/setup.py#L268-L270

[matthew-brett]

Yes, it is a numpy thing, and so a bad example:

      File "numpy/core/setup.py", line 278, in check_types
        "Cannot compile 'Python.h'. Perhaps you need to "
    SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.

Trying with another package, regex:

    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c Python2/_regex.c -o build/temp.linux-x86_64-2.7/Python2/_regex.o
    Python2/_regex.c:46:20: fatal error: Python.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

So - error, arising from gcc not very helpful, but easily solved with a web search. I guess it would make the transition easier, if we could give a more helpful message than that.

[njsmith]

@matthew-brett: Here's a wacky question. What happens if you do patchelf --remove-needed libpython2.7.so.1 multiarray.so?

Reason I ask: For a regular interpreter where the executable's linked against libpython2.7.so.1, it shouldn't matter whether multiarray.so has a DT_NEEDED for libpython2.7.so.1, b/c it will be loaded into the ELF namespace regardless. And for Debian, /usr/bin/python2.7 itself seems to be exporting all the standard symbols... in fact, the multiarray.so in debian's python-numpy package doesn't seem to be linked against libpython at all! And if I build a numpy using debian, the resulting multiarray.so also has no DT_NEEDED for libpython, but isn't statically linked either (objdump shows lots of *UND* symbols for PyWhatever_Foo functions).

OTOH debian doesn't seem to be consistent -- some python-foo packages depend on libpython, and some don't. And I don't know what they've done to distutils so that it doesn't link to -lpython2.7. Something pretty weird going on here :-/

(The one place where failing to link to libpython will break, AFAICT, is if you have a python interpreter that's embedded, but not via DT_NEEDED linking to an executable like gnucash or blender or whatever, but instead the python interpreter itself is dlopen'ed (or linked to another .so that gets dlopen'ed, etc.). In this case the python symbols don't make it into the global ELF namespace, so that won't be visible to extension modules. So AFAICT it really is better if python extensions link to libpython explicitly, and libpython gets included on distros out-of-the-box -- that's how fedora does it. But maybe we can't fix this anytime soon :-/.)

[matthew-brett]

Yup, doing this:

patchelf --remove-needed libpython2.7.so.1.0 /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so
patchelf --remove-needed libpython2.7.so.1.0 /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so
patchelf --remove-needed libpython2.7.so.1.0 /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so 
patchelf --remove-needed libpython2.7.so.1.0 /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so 
patchelf --remove-needed libpython2.7.so.1.0 /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so 
patchelf --remove-needed libpython2.7.so.1.0 /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so 
patchelf --remove-needed libpython2.7.so.1.0 /usr/local/lib/python2.7/dist-packages/numpy/core/tests
patchelf --remove-needed libpython2.7.so.1.0 /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray_tests.so 
patchelf --remove-needed libpython2.7.so.1.0 /usr/local/lib/python2.7/dist-packages/numpy/core/umath_tests.so 
patchelf --remove-needed libpython2.7.so.1.0 /usr/local/lib/python2.7/dist-packages/numpy/core/operand_flag_tests.so 
patchelf --remove-needed libpython2.7.so.1.0 /usr/local/lib/python2.7/dist-packages/numpy/core/struct_ufunc_test.so 

allows the test to run without any missing libraries. Are you suggesting that we do this for manylinux wheels in general?

[njsmith]

I'm really not sure...

[njsmith]

Ah, here's an example Debian-compiled extension module that does link to libpython:

~/src$ readelf -a /usr/lib/python2.7/dist-packages/PySide/QtOpenGL.so | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libpyside-python2.7.so.1.2]
 0x0000000000000001 (NEEDED)             Shared library: [libpython2.7.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libQtCore.so.4]
 0x0000000000000001 (NEEDED)             Shared library: [libQtGui.so.4]
 0x0000000000000001 (NEEDED)             Shared library: [libQtOpenGL.so.4]
 0x0000000000000001 (NEEDED)             Shared library: [libshiboken-python2.7.so.1.2]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

Why? I don't know.

[matthew-brett]

How about floating the idea of editing out the libpython dependency on distutils-sig?

[rmcgibbo]

I'm very nervous about editing it out like that, especially in an automatic tool like auditwheel repair. I don't know exactly what the reason is that some packages get built with a dependency on libpython.so and others don't, or that some packages might have a dependency on libpython.so but seem to work fine when that dependency is removed with patchelf, but it seems like a rash thing to do in general.