pip 7.1.1 broke our install workflow

[adembo]

We have a Python module with the following requirements.txt:

cython >= 0.21
nose >= 1.0
setuptools >= 0.8

The initial set up includes both pip install --upgrade pip and pip install -r requirements.txt, so we're exposed to new versions of pip immediately.

Following the upgrade to pip 7.1.1, the second step has started to fail. Note that we run this on older RHEL6 machines which come with an old version of setuptools out of the box. Here is the error:

Installing collected packages: cython, nose, setuptools
 Running setup.py install for cython
  Found existing installation: setuptools 0.6rc11
    Uninstalling setuptools-0.6rc11:
      Successfully uninstalled setuptools-0.6rc11
Successfully installed cython nose setuptools
Traceback (most recent call last):
  File "/home/adar/python/test_environment/bin/pip", line 8, in <module>
    load_entry_point('pip==7.1.1', 'console_scripts', 'pip')()
  File "/home/adar/python/test_environment/lib/python2.6/site-packages/pip/__init__.py", line 217, > in main
    return command.main(cmd_args)
  File "/home/adar/python/test_environment/lib/python2.6/site-packages/pip/basecommand.py", line 248, in main
    pip_version_check(session)
  File "/home/adar/python/test_environment/lib/python2.6/site-packages/pip/utils/outdated.py", line 102, in pip_version_check
    pip_version = packaging_version.parse(get_installed_version('pip'))
  File "/home/adar/python/test_environment/lib/python2.6/site-packages/pip/utils/__init__.py", line 858, in get_installed_version
    working_set = pkg_resources.WorkingSet()
  File "/home/adar/python/test_environment/lib/python2.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 629, in __init__
    self.add_entry(entry)
  File "/home/adar/python/test_environment/lib/python2.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 685, in add_entry
    for dist in find_distributions(entry, True):
  File "/home/adar/python/test_environment/lib/python2.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2075, in find_eggs_in_zip
    if metadata.has_metadata('PKG-INFO'):
  File "/home/adar/python/test_environment/lib/python2.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1605, in has_metadata
    return self.egg_info and self._has(self._fn(self.egg_info, name))
  File "/home/adar/python/test_environment/lib/python2.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1963, in _has
    return zip_path in self.zipinfo or zip_path in self._index()
  File "/home/adar/python/test_environment/lib/python2.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1843, in zipinfo
    return self._zip_manifests.load(self.loader.archive)
  File "/home/adar/python/test_environment/lib/python2.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1783, in load
    mtime = os.stat(path).st_mtime
OSError: [Errno 2] No such file or directory: '/home/adar/python/test_environment/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg'

I suspect this was due to the move of the pip version check in 7.1.1, and I've successfully worked around it by adding --disable-pip-version-check to the requirements.txt command line, but there's probably still an underlying bug here.

[rbtcollins]

This was due to the removal of the distribute backwards compatibility hack. You'll need to remove that. Something like:
pip uninstall distribute -y || true
pip uninstall setuptools -y || true
pip install setuptools

should do it.

[jraby]

Same problem here. This workflow doesn't work anymore (ubuntu 12.04):

virtualenv  testenv
. testenv/bin/activate

pip install pip==7.1.2
$ pip install -U setuptools
/home/jean/tmp/pouetenv/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
Collecting setuptools
  Using cached setuptools-18.2-py2.py3-none-any.whl
Installing collected packages: setuptools
  Found existing installation: setuptools 0.6rc11
    Uninstalling setuptools-0.6rc11:
      Successfully uninstalled setuptools-0.6rc11
Successfully installed setuptools
Traceback (most recent call last):
  File "/home/jean/tmp/pouetenv/bin/pip", line 8, in <module>
    load_entry_point('pip==7.1.2', 'console_scripts', 'pip')()
  File "/home/jean/tmp/pouetenv/local/lib/python2.7/site-packages/pip/__init__.py", line 217, in main
    return command.main(cmd_args)
  File "/home/jean/tmp/pouetenv/local/lib/python2.7/site-packages/pip/basecommand.py", line 248, in main
    pip_version_check(session)
  File "/home/jean/tmp/pouetenv/local/lib/python2.7/site-packages/pip/utils/outdated.py", line 102, in pip_version_check
    installed_version = get_installed_version("pip")
  File "/home/jean/tmp/pouetenv/local/lib/python2.7/site-packages/pip/utils/__init__.py", line 858, in get_installed_version
    working_set = pkg_resources.WorkingSet()
  File "/home/jean/tmp/pouetenv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 629, in __init__
    self.add_entry(entry)
  File "/home/jean/tmp/pouetenv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 685, in add_entry
    for dist in find_distributions(entry, True):
  File "/home/jean/tmp/pouetenv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2075, in find_eggs_in_zip
    if metadata.has_metadata('PKG-INFO'):
  File "/home/jean/tmp/pouetenv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1605, in has_metadata
    return self.egg_info and self._has(self._fn(self.egg_info, name))
  File "/home/jean/tmp/pouetenv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1963, in _has
    return zip_path in self.zipinfo or zip_path in self._index()
  File "/home/jean/tmp/pouetenv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1843, in zipinfo
    return self._zip_manifests.load(self.loader.archive)
  File "/home/jean/tmp/pouetenv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1783, in load
    mtime = os.stat(path).st_mtime
OSError: [Errno 2] No such file or directory: '/home/jean/tmp/pouetenv/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg'

If I understand correctly, the new workflow would be:

virtualenv testenv
. testenv/bin/activate

pip install -U pip
pip uninstall distribute -y || true
pip uninstall setuptools -y || true
pip install setuptools

It would be kind of cumbersome to have to remember to do this for every virtualenv that needs a recent setuptools, no?

[jraby]

I spoke too fast, uninstalling setuptools as suggested by @rbtcollins breaks pip:

virtualenv testenv
. testenv/bin/activate

pip uninstall distribute
# not installed

pip uninstall setuptools
# errors out as above

$ pip
Traceback (most recent call last):
  File "/home/jean/tmp/testenv/bin/pip", line 5, in <module>
    from pkg_resources import load_entry_point
ImportError: No module named pkg_resources

Until this behaviour is fixed, looks like the only way to make this work is to run pip install -U setuptools in a fresh viertualenv and ignoring the 'no such file or directory' error.
Further calls to pip should work.

[johnyf]

In a fresh virtualenv, I get this error when pip uninstall dd, but not if dd was installed with pip. Only when dd has been installed with python setup.py install. This is the setup.py that causes this behavior. Are there any tips as to what should be changed, in order to avoid this error when python setup.py install is used?

environment:

OS X 10.8.5
python 2.7.10
pip 7.1.2
astutils==0.0.1
dd==0.2.0
decorator==4.0.4
networkx==1.10
ply==3.8
wheel==0.24.0

[rbtcollins]

Always use pip (or another metadata-compatible installer) to install packages. 'setup.py install' is usually not tracking all the metadata correctly.

[johnyf]

I am developing the package, and it has Cython extensions that are compiled, so using python setup.py install or develop is more convenient. I will compare though, to see whether in all cases pip can replace the python setup.py invocation.

When installing from the command line (and not a requirements file), options are available, to fetch and build a C library that the (optional) extension depends on. As described in #1883, these options cannot yet be passed to setup.py via pip, without passing them to the dependent packages, thus raising exceptions. So, there are some cases when pip is less direct (here it would require the --instal-option to be listed in a requirements file).

[johnyf]

A difference between the installations is that pip places dd under the directory ~/.virtualenvs/dev/lib/python2.7/site-packages/dd, whereas python setup.py install in ~/.virtualenvs/dev/lib/python2.7/site-packages/dd-0.2.0-py2.7-macosx-10.4-x86_64.egg. I do not know whether this could explain the error observed.

python setup.py install
...
pip uninstall dd
Uninstalling dd-0.2.0:
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd-0.2.0-py2.7-macosx-10.4-x86_64.egg
Proceed (y/n)? y
  Successfully uninstalled dd-0.2.0
Traceback (most recent call last):
  File "~/.virtualenvs/dev/bin/pip", line 11, in <module>
    sys.exit(main())
  File "~/.virtualenvs/dev/lib/python2.7/site-packages/pip/__init__.py", line 217, in main
    return command.main(cmd_args)
  File "~/.virtualenvs/dev/lib/python2.7/site-packages/pip/basecommand.py", line 248, in main
    pip_version_check(session)
  File "~/.virtualenvs/dev/lib/python2.7/site-packages/pip/utils/outdated.py", line 102, in pip_version_check
    installed_version = get_installed_version("pip")
  File "~/.virtualenvs/dev/lib/python2.7/site-packages/pip/utils/__init__.py", line 858, in get_installed_version
    working_set = pkg_resources.WorkingSet()
  File "~/.virtualenvs/dev/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 629, in __init__
    self.add_entry(entry)
  File "~/.virtualenvs/dev/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 685, in add_entry
    for dist in find_distributions(entry, True):
  File "~/.virtualenvs/dev/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2075, in find_eggs_in_zip
    if metadata.has_metadata('PKG-INFO'):
  File "~/.virtualenvs/dev/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1605, in has_metadata
    return self.egg_info and self._has(self._fn(self.egg_info, name))
  File "~/.virtualenvs/dev/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1963, in _has
    return zip_path in self.zipinfo or zip_path in self._index()
  File "~/.virtualenvs/dev/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1843, in zipinfo
    return self._zip_manifests.load(self.loader.archive)
  File "~/.virtualenvs/dev/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1783, in load
    mtime = os.stat(path).st_mtime
OSError: [Errno 2] No such file or directory: '~/.virtualenvs/dev/lib/python2.7/site-packages/dd-0.2.0-py2.7-macosx-10.4-x86_64.egg'

compared to:

pip install --verbose .
...
pip uninstall dd
Uninstalling dd-0.2.0:
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd-0.2.0-py2.7.egg-info
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/__init__.py
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/__init__.pyc
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/_compat.py
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/_compat.pyc
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/_parser.py
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/_parser.pyc
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/_setup.py
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/_setup.pyc
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/_version.py
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/_version.pyc
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/autoref.py
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/autoref.pyc
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/bdd.py
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/bdd.pyc
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/bdd_parsetab.py
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/bdd_parsetab.pyc
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/cudd.so
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/dddmp.py
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/dddmp.pyc
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/dddmp_parsetab.py
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/dddmp_parsetab.pyc
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/mdd.py
  ~/.virtualenvs/dev/lib/python2.7/site-packages/dd/mdd.pyc
Proceed (y/n)? y
  Successfully uninstalled dd-0.2.0

[dstufft]

Closing this, the way to handle an upgrade breaking you is to pin to an older version or fix the underlying breakage.