Installation with pip fails (on Arch Linux)

[goggle]

I wanted to setup a development build for pandas, but this fails because of an installation error in bottleneck.

First, I set up a virtual environment. Then
pip install bottleneck
gives me this error:

Collecting bottleneck
  Using cached https://files.pythonhosted.org/packages/b8/d9/2b0beb43e0f8b25624699f918d6b00e53c46812da911eb15ec9ffebf1917/Bottleneck-1.3.0.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... error
    ERROR: Command errored out with exit status 1:
     command: /home/alex/Projects/virtualenvs/pandas-dev/bin/python /home/alex/Projects/virtualenvs/pandas-dev/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpfzm0irx4
         cwd: /tmp/pip-install-hvrtun08/bottleneck
    Complete output (75 lines):
    running dist_info
    creating /tmp/pip-modern-metadata-jm_6d26f/Bottleneck.egg-info
    writing /tmp/pip-modern-metadata-jm_6d26f/Bottleneck.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-modern-metadata-jm_6d26f/Bottleneck.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-modern-metadata-jm_6d26f/Bottleneck.egg-info/requires.txt
    writing top-level names to /tmp/pip-modern-metadata-jm_6d26f/Bottleneck.egg-info/top_level.txt
    writing manifest file '/tmp/pip-modern-metadata-jm_6d26f/Bottleneck.egg-info/SOURCES.txt'
    Traceback (most recent call last):
      File "/tmp/pip-build-env-40hp1wwq/overlay/lib/python3.8/site-packages/numpy/core/__init__.py", line 16, in <module>
        from . import multiarray
    ImportError: /tmp/pip-build-env-40hp1wwq/overlay/lib/python3.8/site-packages/numpy/core/multiarray.cpython-38-x86_64-linux-gnu.so: undefined symbol: cblas_cdotc_sub
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/alex/Projects/virtualenvs/pandas-dev/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 257, in <module>
        main()
      File "/home/alex/Projects/virtualenvs/pandas-dev/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 240, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/home/alex/Projects/virtualenvs/pandas-dev/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 110, in prepare_metadata_for_build_wheel
        return hook(metadata_directory, config_settings)
      File "/tmp/pip-build-env-40hp1wwq/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 156, in prepare_metadata_for_build_wheel
        self.run_setup()
      File "/tmp/pip-build-env-40hp1wwq/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 236, in run_setup
        super(_BuildMetaLegacyBackend,
      File "/tmp/pip-build-env-40hp1wwq/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 142, in run_setup
        exec(compile(code, __file__, 'exec'), locals())
      File "setup.py", line 196, in <module>
        setup(**metadata)
      File "/tmp/pip-build-env-40hp1wwq/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib64/python3.8/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib64/python3.8/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/lib64/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/tmp/pip-build-env-40hp1wwq/overlay/lib/python3.8/site-packages/setuptools/command/dist_info.py", line 31, in run
        egg_info.run()
      File "/tmp/pip-build-env-40hp1wwq/overlay/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 296, in run
        self.find_sources()
      File "/tmp/pip-build-env-40hp1wwq/overlay/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 303, in find_sources
        mm.run()
      File "/tmp/pip-build-env-40hp1wwq/overlay/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 534, in run
        self.add_defaults()
      File "/tmp/pip-build-env-40hp1wwq/overlay/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 570, in add_defaults
        sdist.add_defaults(self)
      File "/usr/lib64/python3.8/distutils/command/sdist.py", line 228, in add_defaults
        self._add_defaults_ext()
      File "/usr/lib64/python3.8/distutils/command/sdist.py", line 311, in _add_defaults_ext
        build_ext = self.get_finalized_command('build_ext')
      File "/usr/lib64/python3.8/distutils/cmd.py", line 299, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "/usr/lib64/python3.8/distutils/cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "setup.py", line 75, in finalize_options
        import numpy
      File "/tmp/pip-build-env-40hp1wwq/overlay/lib/python3.8/site-packages/numpy/__init__.py", line 142, in <module>
        from . import add_newdocs
      File "/tmp/pip-build-env-40hp1wwq/overlay/lib/python3.8/site-packages/numpy/add_newdocs.py", line 13, in <module>
        from numpy.lib import add_newdoc
      File "/tmp/pip-build-env-40hp1wwq/overlay/lib/python3.8/site-packages/numpy/lib/__init__.py", line 8, in <module>
        from .type_check import *
      File "/tmp/pip-build-env-40hp1wwq/overlay/lib/python3.8/site-packages/numpy/lib/type_check.py", line 11, in <module>
        import numpy.core.numeric as _nx
      File "/tmp/pip-build-env-40hp1wwq/overlay/lib/python3.8/site-packages/numpy/core/__init__.py", line 26, in <module>
        raise ImportError(msg)
    ImportError:
    Importing the multiarray numpy extension module failed.  Most
    likely you are trying to import a failed build of numpy.
    If you're working with a numpy git repo, try `git clean -xdf` (removes all
    files not under version control).  Otherwise reinstall numpy.
    
    Original error was: /tmp/pip-build-env-40hp1wwq/overlay/lib/python3.8/site-packages/numpy/core/multiarray.cpython-38-x86_64-linux-gnu.so: undefined symbol: cblas_cdotc_sub
    
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/alex/Projects/virtualenvs/pandas-dev/bin/python /home/alex/Projects/virtualenvs/pandas-dev/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpfzm0irx4 Check the logs for full command output.

[qwhelan]

Thanks for reporting - could you provide your pip version and check whether you have a system install of numpy?

[goggle]

Thanks for reporting - could you provide your pip version and check whether you have a system install of numpy?

Sure, Python version is 3.8.0.

❯ pip --version
pip 19.3.1 from /tmp/bottleneck/lib/python3.8/site-packages/pip (python 3.8)

Numpy is installed on the system (version 1.17.4), but cannot be seen by Python in that virtual environment:

❯ python
Python 3.8.0 (default, Oct 23 2019, 18:51:26) 
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'

[qwhelan]

Thanks - that should be a new enough pip.

As for Python not being able to see numpy, gcc can sometimes see the system packages through $PATH or similar variables.

• Is the cblas library visible in your venv?
• If you create a fresh venv and install bottleneck into it, does this error still occur?
• Is there anything else about your environment that I might need in order to reproduce?

[goggle]

Here is a full log of what I did:

❯ cd /tmp/

/tmp 
❯ virtualenv bottleneck
Using base prefix '/usr'
New python executable in /tmp/bottleneck/bin/python
Installing setuptools, pip, wheel...
done.

/tmp took 4s 
❯ . bottleneck/bin/activate.fish

/tmp via 🐍 v3.8.0 (bottleneck) 
❯ pip install bottleneck
Collecting bottleneck
  Using cached https://files.pythonhosted.org/packages/b8/d9/2b0beb43e0f8b25624699f918d6b00e53c46812da911eb15ec9ffebf1917/Bottleneck-1.3.0.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... error
    ERROR: Command errored out with exit status 1:
    .
    .
    .

How do I check for the visibility of cblas in my virtual environment?

[qwhelan]

Thanks for those steps - I've tried them out in Docker with the following Dockerfile:

FROM archlinux
RUN pacman -Sy --noconfirm python-pandas python-virtualenv python-pip gcc python-bottleneck fish
RUN cd /tmp
RUN virtualenv bottleneck
RUN /bin/fish -c ". bottleneck/bin/activate.fish && pip install bottleneck"

Unfortunately, the above works for all combinations I can attempt. I'm not as familiar with Arch so haven't been able to downgrade numpy to below 1.16, which is likely the cause of the issue (possibly combined with an edge case in fish support).

If you're able to upgrade the system python-numpy (and python-bottleneck, if installed), that's likely to fix the issue.

Otherwise, if you're able to tweak the above Dockerfile until the error is reproducible, I can investigate then.

[astrofrog]

I think the issue is that the Python 3.8 case is missing from the pyproject.toml file - see #306 for a fix (I also ran into a similar issue).

[0xb0b]

I have the same error while trying to install fastai on Manjaro (which is based on Arch). bottleneck is the dependency of fastai. It was in a clean virtual environment with Python 3.8.1 and pip 20.0.2. I have openblas installed, and no cblas.
#306 fixes the issue for me. I have managed to install bottleneck from the repo by checking out v1.3.1 and applying the fix:

(python/fastai)> pip install .
 Processing /home/xbob/proj/ml/fastai/bottleneck
   Installing build dependencies ... done
   Getting requirements to build wheel ... done
     Preparing wheel metadata ... done
Collecting numpy
  Using cached numpy-1.18.1-cp38-cp38-manylinux1_x86_64.whl (20.6 MB)
Building wheels for collected packages: Bottleneck
  Building wheel for Bottleneck (PEP 517) ... done
  Created wheel for Bottleneck: filename=Bottleneck-1.3.1+0.g7ac20de.dirty-cp38-cp38-linux_x86_64.whl size=403123 sha256=08ae652823c89303b27dc8c4845a2f0c51d58d602ca0c9cba2bd8c13dbc96f43
  Stored in directory: /tmp/pip-ephem-wheel-cache-5vr1_9d0/wheels/fd/eb/d2/b6ce83911b8c86f5eef36e25df5c75d38d76c8cb70eae63043
Successfully built Bottleneck
Installing collected packages: numpy, Bottleneck
Successfully installed Bottleneck-1.3.1+0.g7ac20de.dirty numpy-1.18.1

[qwhelan]

Thanks for the data point. The thing that has been confusing for me regarding #306 being a potential fix is that pandas has a similar lack of 3.8 being explicitly specified. Perhaps that's being side-stepped by shipping .whls via PyPI.

I may release a 1.3.2 with a patch for this in the next day or two.

[qwhelan]

1.3.2 has been uploaded to PyPI and should be available on conda either later tonight or early tomorrow.

I'm going to close this for now, please feel free to re-open if the new version does not resolve the issue.

[goggle]

Thanks @qwhelan, now installing bottleneck via pip in a virtual environment works without any errors.

[qwhelan]

Fantastic, apologies for the resolution taking so long!

[abhishekshingadiya]

the still same issue facing in pip 22.0.3 with python 3.8

[rdbisme]

@abhishekshingadiya can you provide the steps you followed and the output of the commands in order to have a detailed reproduction? :)