PIP 19.0.2 breaks numpy/pandas install

[cdagraca]

Environment

• pip version: 19.0.2
• Python version: 3.4
• OS: ubuntu

Docker build off base of ubuntu:trusty

Description

Numpy install appears to succeed, but with warnings
Subsequent pandas install can't find numpy

Pinning pip to 19.0.1 changes the error to the issue reported here: #6163
Pinning pip to <19.0 resolves it.

Expected behavior

The packages should install

How to Reproduce

1. Create a "requirements.txt" containing

setuptools
openpyxl
numpy
pandas

2. Create the following Dockerfile:

FROM ubuntu:trusty AS python-packages
RUN apt-get update && \
    apt-get -qy install python3-pip

COPY /config/requirements /tmp/requirements
RUN pip3 install --upgrade pip --ignore-installed -r /tmp/requirements

3. Attempt to build the docker image
4. The error below occurs

Output

Downloading/unpacking numpy (from -r /tmp/requirements (line 3))
  Running setup.py (path:/tmp/pip_build_root/numpy/setup.py) egg_info for package numpy
    Running from numpy source directory.
    /usr/lib/python3.4/distutils/dist.py:260: UserWarning: Unknown distribution option: 'python_requires'
      warnings.warn(msg)

    no previously-included directories found matching 'doc/build'
    no previously-included directories found matching 'doc/source/generated'
    no previously-included directories found matching 'benchmarks/env'
    no previously-included directories found matching 'benchmarks/results'
    no previously-included directories found matching 'benchmarks/html'
    no previously-included directories found matching 'benchmarks/numpy'
    no previously-included directories found matching '*/__pycache__'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '*.pyo' found anywhere in distribution
    warning: no previously-included files matching '*.pyd' found anywhere in distribution
    warning: no previously-included files matching '*.swp' found anywhere in distribution
    warning: no previously-included files matching '*.bak' found anywhere in distribution
    warning: no previously-included files matching '*~' found anywhere in distribution
Downloading/unpacking pandas (from -r /tmp/requirements (line 4))
  Running setup.py (path:/tmp/pip_build_root/pandas/setup.py) egg_info for package pandas
    Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/pkg_resources.py", line 225, in get_provider
        module = sys.modules[moduleOrReq]
    KeyError: 'numpy'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/tmp/pip_build_root/pandas/setup.py", line 732, in <module>
        ext_modules=maybe_cythonize(extensions, compiler_directives=directives),
      File "/tmp/pip_build_root/pandas/setup.py", line 475, in maybe_cythonize
        numpy_incl = pkg_resources.resource_filename('numpy', 'core/include')
      File "/usr/lib/python3/dist-packages/pkg_resources.py", line 935, in resource_filename
        return get_provider(package_or_requirement).get_resource_filename(
      File "/usr/lib/python3/dist-packages/pkg_resources.py", line 227, in get_provider
        __import__(moduleOrReq)
    ImportError: No module named 'numpy'
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "/usr/lib/python3/dist-packages/pkg_resources.py", line 225, in get_provider

    module = sys.modules[moduleOrReq]

KeyError: 'numpy'

[huntekah]

same issue here:
my requirements has numpy and pandas

[cjerdonek]

@ncoghlan Any idea here since you worked on the fix for #6163?

[cjerdonek]

@cdagraca Do the warnings also occur with < 19? Can you include what the successful output looks like for comparison?

[cdagraca]

@cjerdonek output pinning PIP < 19.0 is significantly different - those warning are not there, but it's not directly comparable:

Collecting numpy (from -r /tmp/requirements (line 3))
  Downloading https://files.pythonhosted.org/packages/2b/26/07472b0de91851b6656cbc86e2f0d5d3a3128e7580f23295ef58b6862d6c/numpy-1.16.1.zip (5.1MB)
Collecting pandas (from -r /tmp/requirements (line 4))
  Downloading https://files.pythonhosted.org/packages/08/01/803834bc8a4e708aedebb133095a88a4dad9f45bbaf5ad777d2bea543c7e/pandas-0.22.0.tar.gz (11.3MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
Collecting xlsxwriter (from -r /tmp/requirements (line 5))
  Downloading https://files.pythonhosted.org/packages/8d/d0/f4ee9c621902d18807dd88b0a0ff4426666af8dd1e952c4943a9415f3eeb/XlsxWriter-1.1.4-py2.py3-none-any.whl (142kB)
Collecting IPython (from -r /tmp/requirements (line 6))
  Downloading https://files.pythonhosted.org/packages/f7/62/2fef7db3a7b75e8099c3d9db2630ae5ba0b9eefefd91f7497862393d90e8/ipython-6.5.0-py3-none-any.whl (748kB)
Collecting jdcal (from openpyxl->-r /tmp/requirements (line 2))
  Downloading https://files.pythonhosted.org/packages/a0/38/dcf83532480f25284f3ef13f8ed63e03c58a65c9d3ba2a6a894ed9497207/jdcal-1.4-py2.py3-none-any.whl
Collecting et_xmlfile (from openpyxl->-r /tmp/requirements (line 2))
  Downloading https://files.pythonhosted.org/packages/22/28/a99c42aea746e18382ad9fb36f64c1c1f04216f41797f2f0fa567da11388/et_xmlfile-1.0.1.tar.gz
Collecting python-dateutil>=2 (from pandas->-r /tmp/requirements (line 4))
  Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)
Collecting pytz>=2011k (from pandas->-r /tmp/requirements (line 4))
  Downloading https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl (510kB)
Collecting pickleshare (from IPython->-r /tmp/requirements (line 6))
  Downloading https://files.pythonhosted.org/packages/9a/41/220f49aaea88bc6fa6cba8d05ecf24676326156c23b991e80b3f2fc24c77/pickleshare-0.7.5-py2.py3-none-any.whl
Collecting traitlets>=4.2 (from IPython->-r /tmp/requirements (line 6))
  Downloading https://files.pythonhosted.org/packages/93/d6/abcb22de61d78e2fc3959c964628a5771e47e7cc60d53e9342e21ed6cc9a/traitlets-4.3.2-py2.py3-none-any.whl (74kB)
Collecting simplegeneric>0.8 (from IPython->-r /tmp/requirements (line 6))
  Downloading https://files.pythonhosted.org/packages/3d/57/4d9c9e3ae9a255cd4e1106bb57e24056d3d0709fc01b2e3e345898e49d5b/simplegeneric-0.8.1.zip
Collecting typing; python_version <= "3.4" (from IPython->-r /tmp/requirements (line 6))
  Downloading https://files.pythonhosted.org/packages/4a/bd/eee1157fc2d8514970b345d69cb9975dcd1e42cd7e61146ed841f6e68309/typing-3.6.6-py3-none-any.whl
Collecting backcall (from IPython->-r /tmp/requirements (line 6))
  Downloading https://files.pythonhosted.org/packages/84/71/c8ca4f5bb1e08401b916c68003acf0a0655df935d74d93bf3f3364b310e0/backcall-0.1.0.tar.gz
Collecting prompt-toolkit<2.0.0,>=1.0.15 (from IPython->-r /tmp/requirements (line 6))
  Downloading https://files.pythonhosted.org/packages/04/d1/c6616dd03701e7e2073f06d5c3b41b012256e42b72561f16a7bd86dd7b43/prompt_toolkit-1.0.15-py3-none-any.whl (247kB)
Collecting pexpect; sys_platform != "win32" (from IPython->-r /tmp/requirements (line 6))
  Downloading https://files.pythonhosted.org/packages/89/e6/b5a1de8b0cc4e07ca1b305a4fcc3f9806025c1b651ea302646341222f88b/pexpect-4.6.0-py2.py3-none-any.whl (57kB)
Collecting jedi>=0.10 (from IPython->-r /tmp/requirements (line 6))
  Downloading https://files.pythonhosted.org/packages/c2/bc/54d53f5bc4658380d0eca9055d72be4df45e5bfd91a4bac97da224a92553/jedi-0.13.2-py2.py3-none-any.whl (177kB)
Collecting decorator (from IPython->-r /tmp/requirements (line 6))
  Downloading https://files.pythonhosted.org/packages/f1/cd/7c8240007e9716b14679bc217a1baefa4432aa30394f7e2ec40a52b1a708/decorator-4.3.2-py2.py3-none-any.whl
Collecting pygments (from IPython->-r /tmp/requirements (line 6))
  Downloading https://files.pythonhosted.org/packages/13/e5/6d710c9cf96c31ac82657bcfb441df328b22df8564d58d0c4cd62612674c/Pygments-2.3.1-py2.py3-none-any.whl (849kB)
Collecting six>=1.5 (from python-dateutil>=2->pandas->-r /tmp/requirements (line 4))
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting ipython-genutils (from traitlets>=4.2->IPython->-r /tmp/requirements (line 6))
  Downloading https://files.pythonhosted.org/packages/fa/bc/9bd3b5c2b4774d5f33b2d544f1460be9df7df2fe42f352135381c347c69a/ipython_genutils-0.2.0-py2.py3-none-any.whl
Collecting wcwidth (from prompt-toolkit<2.0.0,>=1.0.15->IPython->-r /tmp/requirements (line 6))
  Downloading https://files.pythonhosted.org/packages/7e/9f/526a6947247599b084ee5232e4f9190a38f398d7300d866af3ab571a5bfe/wcwidth-0.1.7-py2.py3-none-any.whl
Collecting ptyprocess>=0.5 (from pexpect; sys_platform != "win32"->IPython->-r /tmp/requirements (line 6))
  Downloading https://files.pythonhosted.org/packages/d1/29/605c2cc68a9992d18dada28206eeada56ea4bd07a239669da41674648b6f/ptyprocess-0.6.0-py2.py3-none-any.whl
Collecting parso>=0.3.0 (from jedi>=0.10->IPython->-r /tmp/requirements (line 6))
  Downloading https://files.pythonhosted.org/packages/19/b1/522b2671cc6d134c9d3f5dfc0d02fee07cab848e908d03d2bffea78cca8f/parso-0.3.4-py2.py3-none-any.whl (93kB)
Building wheels for collected packages: openpyxl, numpy, pandas, et-xmlfile, simplegeneric, backcall
  Running setup.py bdist_wheel for openpyxl: started
  Running setup.py bdist_wheel for openpyxl: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/ec/57/8e/585318833ce01a0181c32e9676e0321074dcc238c8843c8622
  Running setup.py bdist_wheel for numpy: started
  Running setup.py bdist_wheel for numpy: still running...
  Running setup.py bdist_wheel for numpy: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/04/64/e1/283a3672c2865608968594c02a6923311f44d033bcece2683b
  Running setup.py bdist_wheel for pandas: started
  Running setup.py bdist_wheel for pandas: still running...
  Running setup.py bdist_wheel for pandas: still running...
  Running setup.py bdist_wheel for pandas: still running...
  Running setup.py bdist_wheel for pandas: still running...
  Running setup.py bdist_wheel for pandas: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/21/6e/f3/24ca4e2d50a98d3733ca82c0e0941d3b411b3212aacf7d9a73
  Running setup.py bdist_wheel for et-xmlfile: started
  Running setup.py bdist_wheel for et-xmlfile: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/2a/77/35/0da0965a057698121fc7d8c5a7a9955cdbfb3cc4e2423cad39
  Running setup.py bdist_wheel for simplegeneric: started
  Running setup.py bdist_wheel for simplegeneric: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/a9/28/53/f24776b4c5bcbe91aaf1f1e247bd6fadd17191aa12fac63902
  Running setup.py bdist_wheel for backcall: started
  Running setup.py bdist_wheel for backcall: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/98/b0/dd/29e28ff615af3dda4c67cab719dd51357597eabff926976b45
Successfully built openpyxl numpy pandas et-xmlfile simplegeneric backcall
Installing collected packages: setuptools, jdcal, et-xmlfile, openpyxl, numpy, six, python-dateutil, pytz, pandas, xlsxwriter, pickleshare, decorator, ipython-genutils, traitlets, simplegeneric, typing, backcall, wcwidth, prompt-toolkit, ptyprocess, pexpect, parso, jedi, pygments, IPython
Successfully installed IPython-6.5.0 backcall-0.1.0 decorator-4.3.2 et-xmlfile-1.0.1 ipython-genutils-0.2.0 jdcal-1.4 jedi-0.13.2 numpy-1.16.1 openpyxl-2.5.14 pandas-0.22.0 parso-0.3.4 pexpect-4.6.0 pickleshare-0.7.5 prompt-toolkit-1.0.15 ptyprocess-0.6.0 pygments-2.3.1 python-dateutil-2.8.0 pytz-2018.9 setuptools-40.8.0 simplegeneric-0.8.1 six-1.12.0 traitlets-4.3.2 typing-3.6.6 wcwidth-0.1.7 xlsxwriter-1.1.4

[daa]

It seems that you are using old pip to install modern numpy and pandas. Try to check it with pip3 --version. Following command does not mean that pip will be upgraded before processing requirements, instead pip will install everything given in one step:

pip3 install --upgrade pip --ignore-installed -r /tmp/requirements

Try first upgrade pip: pip3 install --upgrade pip and then install requirements: pip3.4 install --ignore-installed -r /tmp/requirements.

I tried and got following successful installation log:

 pip3.4 install --upgrade --ignore-installed -r r.txt 
DEPRECATION: Python 3.4 support has been deprecated. pip 19.1 will be the last one supporting it. Please upgrade your Python as Python 3.4 won't be maintained after March 2019 (cf PEP 429).
Collecting setuptools (from -r r.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/d1/6a/4b2fcefd2ea0868810e92d519dacac1ddc64a2e53ba9e3422c3b62b378a6/setuptools-40.8.0-py2.py3-none-any.whl (575kB)
    100% |################################| 583kB 26.3MB/s 
Collecting numpy (from -r r.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/2b/26/07472b0de91851b6656cbc86e2f0d5d3a3128e7580f23295ef58b6862d6c/numpy-1.16.1.zip
Collecting pandas (from -r r.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/08/01/803834bc8a4e708aedebb133095a88a4dad9f45bbaf5ad777d2bea543c7e/pandas-0.22.0.tar.gz (11.3MB)
    100% |################################| 11.3MB 3.1MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
Collecting pytz>=2011k (from pandas->-r r.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl (510kB)
    100% |################################| 512kB 30.1MB/s 
Collecting python-dateutil>=2 (from pandas->-r r.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)
    100% |################################| 235kB 37.0MB/s 
Collecting six>=1.5 (from python-dateutil>=2->pandas->-r r.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Building wheels for collected packages: pandas
  Building wheel for pandas (PEP 517) ... done
  Stored in directory: /root/.cache/pip/wheels/21/6e/f3/24ca4e2d50a98d3733ca82c0e0941d3b411b3212aacf7d9a73
Successfully built pandas
Building wheels for collected packages: numpy
  Building wheel for numpy (setup.py) ... done
  Stored in directory: /root/.cache/pip/wheels/04/64/e1/283a3672c2865608968594c02a6923311f44d033bcece2683b
Successfully built numpy
Installing collected packages: setuptools, numpy, pytz, six, python-dateutil, pandas
Successfully installed numpy-1.16.1 pandas-0.22.0 python-dateutil-2.8.0 pytz-2018.9 setuptools-40.8.0 six-1.12.0

Also if you'd upgrade Python it will be possible to use pre-built wheels for numpy and pandas and to use recent pandas instead of 0.22.0.

[pfmoore]

@cdagraca as @daa says in your docker script, you're upgrading (the system) pip and installing the other requirements all in a single command. That doesn't guarantee which version of pip will be getting used to install the other requirements.

Do you still see the error if you split the pip upgrade and the requirements install into separate steps?

RUN pip3 install --upgrade pip
RUN pip3 install -r /tmp/requirements

[cdagraca]

@pfmoore I have tried it both ways. Splitting the install as you suggest still results in

Downloading/unpacking numpy (from -r /tmp/requirements (line 3))
  Running setup.py (path:/tmp/pip_build_root/numpy/setup.py) egg_info for package numpy
    Running from numpy source directory.
    /usr/lib/python3.4/distutils/dist.py:260: UserWarning: Unknown distribution option: 'python_requires'
      warnings.warn(msg)

    no previously-included directories found matching 'doc/build'
    no previously-included directories found matching 'doc/source/generated'
    no previously-included directories found matching 'benchmarks/env'
    no previously-included directories found matching 'benchmarks/results'
    no previously-included directories found matching 'benchmarks/html'
    no previously-included directories found matching 'benchmarks/numpy'
    no previously-included directories found matching '*/__pycache__'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '*.pyo' found anywhere in distribution
    warning: no previously-included files matching '*.pyd' found anywhere in distribution
    warning: no previously-included files matching '*.swp' found anywhere in distribution
    warning: no previously-included files matching '*.bak' found anywhere in distribution
    warning: no previously-included files matching '*~' found anywhere in distribution
Downloading/unpacking pandas (from -r /tmp/requirements (line 4))
  Running setup.py (path:/tmp/pip_build_root/pandas/setup.py) egg_info for package pandas
    Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/pkg_resources.py", line 225, in get_provider
        module = sys.modules[moduleOrReq]
    KeyError: 'numpy'

[daa]

Try to use pip3.4, pip3 is still the old pip after upgrade.

[ncoghlan]

Since this is Debian and the Docker file is attempting to overwrite the system pip with a pip-installed one, this has a high chance of being a modified symptom of the fact that doing that isn't supported by either pip or Debian problem: #5599

Especially suspicious is the warning about python_requires not being supported, which suggests an ancient setuptools is getting invoked somewhere.

[pradyunsg]

Splitting the install as you suggest still results in

Right, that isn't a pip bug. There's an issue opened in pandas's repository for the same error message. pandas-dev/pandas#25193

I'm inclined to go ahead and close this issue, since the 2 problems here (upgrading pip simultaneously w/ other packages & the pandas build bug) aren't something that have any actionable things from pip's POV.