Missing dependencies when using python:3

[pslacerda]

I keep my images with the tag python:3 because Python versioning says that 3.x are always compatible with previous 3.x versions. However latest update on tag python:3 (that now is 3.7.0-stretch) causes build failures on my requirements.txt.

The headers or library files could not be found for zlib,
a required dependency when compiling Pillow from source.

Because of this 3.7 was not ready to leave rc.

The workaround is simply change base image from python:3 to python:3.6.

[tao12345666333]

I have a test.

(37) ➜  ~ sudo docker run -it --rm python:3.7.0-stretch /bin/bash
root@3dfb50eb45aa:/#
root@3dfb50eb45aa:/# cat /etc/os-release                                                                                                
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"                                                                                              
NAME="Debian GNU/Linux"                                                                           
VERSION_ID="9"                                                         
VERSION="9 (stretch)"                                                               
ID=debian                                                                                
HOME_URL="https://www.debian.org/"                                                       
SUPPORT_URL="https://www.debian.org/support"                                            
BUG_REPORT_URL="https://bugs.debian.org/"              
root@3dfb50eb45aa:/# pip install Pillow
Collecting Pillow
  Downloading https://files.pythonhosted.org/packages/6e/5d/af4af9a0f57ce81e6710836883986c2ea40b96579f7bd90dc7c1fa375355/Pillow-5.2.0-cp
37-cp37m-manylinux1_x86_64.whl (2.0MB)
    100% |████████████████████████████████| 2.0MB 211kB/s
Installing collected packages: Pillow
Successfully installed Pillow-5.2.0
root@3dfb50eb45aa:/# pip install https://files.pythonhosted.org/packages/6e/5d/af4af9a0f57ce81e6710836883986c2ea40b96579f7bd90dc7c1fa375
355/Pillow-5.2.0-cp37-cp37m-manylinux1_x86_64.whl
Requirement already satisfied: Pillow==5.2.0 from https://files.pythonhosted.org/packages/6e/5d/af4af9a0f57ce81e6710836883986c2ea40b9657
9f7bd90dc7c1fa375355/Pillow-5.2.0-cp37-cp37m-manylinux1_x86_64.whl in /usr/local/lib/python3.7/site-packages (5.2.0)
root@3dfb50eb45aa:/#

It can install successfuly by pip.

And I install Pillow by source code.

root@3dfb50eb45aa:/# wget https://files.pythonhosted.org/packages/d3/c4/b45b9c0d549f482dd072055e2d3ced88f3b977f7b87c7a990228b20e7da1/Pil
low-5.2.0.tar.gz                                                                                                                        
--2018-07-07 10:44:14--  https://files.pythonhosted.org/packages/d3/c4/b45b9c0d549f482dd072055e2d3ced88f3b977f7b87c7a990228b20e7da1/Pill
ow-5.2.0.tar.gz                                                                                                                         
Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.197.63, 2a04:4e42:2e::319                                          
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.197.63|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14499068 (14M) [binary/octet-stream]
Saving to: ‘Pillow-5.2.0.tar.gz’

Pillow-5.2.0.tar.gz               100%[=============================================================>]  13.83M   211KB/s    in 1m 45s

2018-07-07 10:46:00 (135 KB/s) - ‘Pillow-5.2.0.tar.gz’ saved [14499068/14499068]


root@3dfb50eb45aa:/# pip install Pillow-5.2.0.tar.gz                                                                                    
Processing /Pillow-5.2.0.tar.gz                                      
Building wheels for collected packages: Pillow                            
  Running setup.py bdist_wheel for Pillow ... error                  
  Complete output from command /usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-_pl57v2h/setup.py';
f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_w
heel -d /tmp/pip-wheel-_ztydilp --python-tag cp37:                       
  running bdist_wheel                                                 
  running build                                                                 
  running build_py                                                    
  creating build                                                               
  creating build/lib.linux-x86_64-3.7                                   
  creating build/lib.linux-x86_64-3.7/PIL                       
  ...(ignore)

      During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-_pl57v2h/setup.py", line 807, in <module>
        raise RequiredDependencyException(msg)
    __main__.RequiredDependencyException:

    The headers or library files could not be found for zlib,
    a required dependency when compiling Pillow from source.

    Please see the install instructions at:
       https://pillow.readthedocs.io/en/latest/installation.html

I got the seem issue.

But, let's look at the installed packages.

root@3dfb50eb45aa:/# dpkg-query -l | grep zlib                                                                                  
ii  zlib1g:amd64                       1:1.2.8.dfsg-5                 amd64        compression library - runtime                      
ii  zlib1g-dev:amd64                   1:1.2.8.dfsg-5                 amd64        compression library - development          

zlib and zlib-dev has been installed.

The Pillow's setup.py needs zlib.h.

The actual path is /usr/include/zlib.h.

root@1c2b60b90fff:/# dpkg -L zlib1g-dev:amd64 |grep zlib.h
/usr/include/zlib.h
/usr/share/doc/zlib1g-dev/examples/zlib_how.html

The source code https://github.com/python-pillow/Pillow/blob/5.2.x/setup.py#L452 return True, but next line reture None.

So, https://github.com/python-pillow/Pillow/blob/5.2.x/setup.py#L582 raised.

[pslacerda]

Yes, some symlinks (eg libz.so -> libz.so.1) are missing and also the command dpkg-architecture that is called in https://github.com/python-pillow/Pillow/blob/5.2.x/setup.py#L736 to determine library directories.

Edit:
ln /lib/x86_64-linux-gnu/{libz.so.1,libz.so} && apt update && apt install -y dpkg-dev resolved the issue.

I think that the problem is caused by this commit debuerreotype/docker-debian-artifacts@a6d5897.

[tao12345666333]

Yes, so it's not the image's issue, dpkg-dev package is not Python's runtime dependency.

[pslacerda]

I agree. Probably is more related to Pillow (because of dpkg-architecture) and/or debuerreotype (because of missing libz.so -> libz.so.1) and the image is in between.

[pslacerda]

So I'm closing this issue.

[pslacerda]

Answering the question:

Pillow needs dpkg-archtecture that was available in python:3 but is not anymore. Installing dpkg-dev resolve the issue.

[yosifkit]

dpkg-dev has not existed in the python:3 image since cd1f11a. This is an oversight on our part and we will look to change the build dependency installation to be more like docker-library/php#556 so that this won't happen in the future.

As far as the missing symlink for libz.so.1, that is a problem with your tool not using the ld config to find where to look in Debian in general. Debian installs them to /usr/lib/x86_64-linux-gnu/libz.so and /lib/x86_64-linux-gnu/libz.so.1 via the zlib1g-dev and zlib1g packages, respectively (assuming x86-64 architecture). And both packages are installed the python:3.7-stretch and 3.6-stretch images:

$ docker run --rm python:3.7-stretch dpkg -l | grep zlib
ii  zlib1g:amd64                       1:1.2.8.dfsg-5                 amd64        compression library - runtime
ii  zlib1g-dev:amd64                   1:1.2.8.dfsg-5                 amd64        compression library - development
$ docker run --rm python:3.6-stretch dpkg -l | grep zlib
ii  zlib1g:amd64                       1:1.2.8.dfsg-5                 amd64        compression library - runtime
ii  zlib1g-dev:amd64                   1:1.2.8.dfsg-5                 amd64        compression library - development

[pslacerda]

Glad that you fixed!

About the symlink, it was my mistake. I was looking for it on the wrong directories, but ldconfig -p correctly shows where it is installed.

Can I ask you what about docker-library/php#556 do you plan to do? Save the list of packages installed on previous stages, install what is needed for the build and then autoremove them. There is something about it that I missed?

As curiosity, do you know if can be assumed that if ldconfig is provided by glibc then the cache will be at /etc/ld.so.cache and if the binary contents are stable across glibc versions and linux distributions? I know that it seems to holds true at Debian and Fedora, but I'm not sure about other distributions.