Unexplainable ContextualVersionConflict

[1313e]

Not sure how I am going to describe this, as I cannot find any way to reproduce this on my own machine, but for some reason, on Travis CI for Python 2.7.15, pip fails to uninstall the already present numpy v1.16.0 and install the requested numpy<1.16.0,>=1.12.0.
The build job can be viewed here: https://travis-ci.com/1313e/PRISM/jobs/178432270

I know that when running on Travis CI on Linux, it automatically comes with NumPy 1.16.0, which is incompatible with what I want to test for Python 2.7.
Therefore, when installing all requirements, it should uninstall that version and install a version that satisfies the condition (which is v1.15.4).
Until today, it did this perfectly fine and it still does on my machine(s).
However, for some odd reason, it does not do that anymore and simply throws an ContextualVersionConflict.

The only thing I changed today is that I combined two requirements files into a single one.
One has a package requirement that simply requires NumPy (emcee), while the other has the version restrictions.
This does work perfectly fine on my machine(s), even in fresh environments, so I am a bit confused why this pops up all of a sudden.

I know this may seem a bit of a weird issue, as I cannot give any way or method to reproduce the error, but I hope somebody knows what the problem is here.

Edit: It seems that this problem affects both pip v18.1 and v19.0.2.

[pradyunsg]

Does this also occur with pip 19.0.2 and updated setuptools / wheel?

[1313e]

@pradyunsg Yup, it occurs with that as well: https://travis-ci.com/1313e/PRISM/jobs/178462602

[cjerdonek]

Never mind, I see from the Travis CI that this occurred with pip 18.1

[1313e]

@cjerdonek What do you mean?
In the second job output I showed, the same error was produced while using pip v19.0.2.

[cjerdonek]

What happened is that I couldn't tell from your original report without digging into the Travis CI details whether your issue was limited to pip 19.0 and up. But digging in I saw the first report was about 18.1. (It's always a good idea to include the affected pip versions in the issue report text itself.)

[1313e]

Alright, yeah, forgot about that.
It happens for both v18.1 and v19.0.2.

[uranusjr]

I did some digging, and this seems to happen only when you

• Use Travis’s default environment
• Have an existing installation of package X
• Perform an editable install of package Y requiring an incompatible X

This is the minimal reproducible case I came up with:

# setup.py
from setuptools import setup

setup(
    name='pip-conflict-travis',
    install_requires=['six>=1.0.0,<1.11.0'],
)

# Failing .travis.yml
language: python
python: '2.7'

install:
  - python -m pip install six==1.11.0
  - python -m pip install -e .

script:
  -

As mentioned previously, this is only reproducible on Travis. The problem also does not exist if you create your own virtual environment; the following works:

# Working .travis.yml
language: python
python: '2.7'

install:
  - virtualenv --python=$(which python2.7) venv
  - ./venv/bin/pip install six==1.11.0
  - ./venv/bin/pip install -e .

script:
  -

So I am inclined to think this is a configuration on Travis’s part, and there’s not much pip can do at the moment.

[uranusjr]

Forgot to mention—The same failure can be observed on Travis with Ubuntu Trusty and pip 9.0.1 (the default runtime), so it is highly unlikely to be related to any recent packaging changes.

[1313e]

@uranusjr Interesting.
I already had the feeling it was Travis that was causing this problem, but nevertheless, it still feels to me like pip has something to do with it as well.
After all, why would it fail to parse in the requirement properly?

Also, before, I did not have this problem.
The only difference is that now pip will check for both requirements at the same time, while before they were done separately (two pip install -r requirements.txt calls instead of one containing the other).

Forgot to mention—The same failure can be observed on Travis with Ubuntu Trusty and pip 9.0.1 (the default runtime), so it is highly unlikely to be related to any recent packaging changes.

I agree, but it might be an indication that there is a long-living problem somewhere.

[cjerdonek]

Some suggestions:

• What happens if you run pip freeze in the Travis environment both before and after the pip install six==1.11.0 line?
• What do you see when you enable more verbose logging?
• What directory is Travis installing packages into?
• Does this also affect 3.x versions on Travis?

[uranusjr]

pip does not fail to parse the requirement. The failure means “the "best" so far conflicts with a dependency” (quoting the comment where the exception is raised; I have no idea what that means).

I plan to continue digging and see if I can find what causes this.

[pfmoore]

You're using an Ubuntu distribution, I believe, and the system provided pip? If so, then could it be that the Ubuntu/Debian's patches to pip to are the issue here (in particular, I'd be concerned about their override to make --ignore-installed and --user the default)?

[uranusjr]

@pfmoore It is definitely not the pip provided by apt’s python-pip, and Travis doc says it’s a virtualenv. I don’t know whether and how it’s different from a virtualenv created manually though.

[uranusjr]

Update:

I realised that six==1.11.0 was installed by default (like numpy) in Travis’s virtualenv, so I switched to use scripttest instead. The results are the same.

• This also happens on Python 3.6, and likely other Python versions as well.
• Output of pip install -vvv. Not particularly helpful.
• Packages are installed into /home/travis/virtualenv/python$PY_XYZ/lib/python$PY_XY/site-packages, as expected. This is an isolated virtualenv chosen based on the specified Python version, according to Travis documentation.

Environment before installing scripttest:

$ python -m pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
attrs (17.3.0)
mock (2.0.0)
nose (1.3.7)
numpy (1.13.3)
pbr (3.1.1)
pip (9.0.1)
pluggy (0.6.0)
py (1.5.2)
pytest (3.3.0)
setuptools (38.2.4)
six (1.11.0)
wheel (0.30.0)

$ python -m pip freeze
attrs==17.3.0
mock==2.0.0
nose==1.3.7
numpy==1.13.3
pbr==3.1.1
pluggy==0.6.0
py==1.5.2
pytest==3.3.0
six==1.11.0

$ ls $(python -c 'from distutils.sysconfig import get_python_lib; print(get_python_lib())')
attr			pbr-3.1.1.dist-info	pytest-3.3.0.dist-info
attrs-17.3.0.dist-info	pip			pytest.py
easy_install.py		pip-9.0.1.dist-info	setuptools
mock			pkg_resources		setuptools-38.2.3.dist-info
mock-2.0.0.dist-info	pluggy			setuptools-38.2.4.dist-info
nose			pluggy-0.6.0.dist-info	six-1.11.0.dist-info
nose-1.3.7.dist-info	py			six.py
numpy			py-1.5.2.dist-info	wheel
numpy-1.13.3.dist-info	__pycache__		wheel-0.30.0.dist-info
pbr			_pytest

After installing scripttest (and before pip install -e .):

$ python -m pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
attrs (17.3.0)
mock (2.0.0)
nose (1.3.7)
numpy (1.13.3)
pbr (3.1.1)
pip (9.0.1)
pluggy (0.6.0)
py (1.5.2)
pytest (3.3.0)
scripttest (1.3)
setuptools (38.2.4)
six (1.11.0)
wheel (0.30.0)

$ python -m pip freeze
attrs==17.3.0
mock==2.0.0
nose==1.3.7
numpy==1.13.3
pbr==3.1.1
pluggy==0.6.0
py==1.5.2
pytest==3.3.0
scripttest==1.3
six==1.11.0

$ ls $(python -c 'from distutils.sysconfig import get_python_lib; print(get_python_lib())')
attr			pip			scripttest-1.3.dist-info
attrs-17.3.0.dist-info	pip-9.0.1.dist-info	scripttest.py
easy_install.py		pkg_resources		setuptools
mock			pluggy			setuptools-38.2.3.dist-info
mock-2.0.0.dist-info	pluggy-0.6.0.dist-info	setuptools-38.2.4.dist-info
nose			py			six-1.11.0.dist-info
nose-1.3.7.dist-info	py-1.5.2.dist-info	six.py
numpy			__pycache__		wheel
numpy-1.13.3.dist-info	_pytest			wheel-0.30.0.dist-info
pbr			pytest-3.3.0.dist-info
pbr-3.1.1.dist-info	pytest.py

Interestingly, there are two .dist-info database for setuptools, but that doesn’t seem to be related. (I tried deleting the stray dist-info directory; does not change the outcome.)

[cjerdonek]

What happens if you try to reproduce the issue locally (or in a fresh virtualenv) starting with those same packages installed?

[uranusjr]

@cjerdonek Can’t reproduce locally, installation ends successfully. This gave me an idea, however—I tried to reproduce this with a fresh virtualenv on Travis, and voila! It fails.

Next step: Try to build an environment more similar to Travis’s 🤔

[uranusjr]

I think I’ve found the problem. This happens when you try to build an egg-info with a conflicting requirement in the environment.

# setup.py
from setuptools import setup
setup(
    name='conflict-test',
    install_requires=['scripttest!=1.3'],
)

# IMPORTANT: Trigger egg-info rebuild.
rm -rf ./conflict_test.egg-info

# Build a fresh environment.
rm -rf ./venv
python3.6 -m venv venv
PYTHON="$PWD/venv/bin/python"

# Install an incompatible requirement.
$PYTHON -m pip install scripttest==1.3

# FAILURE!
$PYTHON -m pip install -e .

When running pip install, pip tries to check that the metadata is ready with InstallRequirement.check_if_exists(). For an sdist/editable (I think, please correct me if I’m wrong), this means it looks for the egg-info directory, and sets context whether the requirement is satisfied or has conflicts.

pip/src/pip/_internal/req/req_install.py

Lines 376 to 416 in cba6083

	def check_if_exists(self, use_user_site):
	# type: (bool) -> bool
	"""Find an installed distribution that satisfies or conflicts
	with this requirement, and set self.satisfied_by or
	self.conflicts_with appropriately.
	"""
	if self.req is None:
	return False
	try:
	# get_distribution() will resolve the entire list of requirements
	# anyway, and we've already determined that we need the requirement
	# in question, so strip the marker so that we don't try to
	# evaluate it.
	no_marker = Requirement(str(self.req))
	no_marker.marker = None
	self.satisfied_by = pkg_resources.get_distribution(str(no_marker))
	if self.editable and self.satisfied_by:
	self.conflicts_with = self.satisfied_by
	# when installing editables, nothing pre-existing should ever
	# satisfy
	self.satisfied_by = None
	return True
	except pkg_resources.DistributionNotFound:
	return False
	except pkg_resources.VersionConflict:
	existing_dist = pkg_resources.get_distribution(
	self.req.name
	)
	if use_user_site:
	if dist_in_usersite(existing_dist):
	self.conflicts_with = existing_dist
	elif (running_under_virtualenv() and
	dist_in_site_packages(existing_dist)):
	raise InstallationError(
	"Will not install to the user site because it will "
	"lack sys.path precedence to %s in %s" %
	(existing_dist.project_name, existing_dist.location)
	)
	else:
	self.conflicts_with = existing_dist
	return True

From what I understand, the large try-except block here follows this logic:

1. Try to find an existing, compatible distribution for the package to use.
2. If no distributions are found, good (we’ll install it).
3. If a distribution is found, but it is incompatible, record the offending distribution.

The problem is how 3. is implemented. The code assumes that if get_distribution("{package}=={version}") fails with VersionConflict, get_distribution("{package}") (dropping the specifiers) should return the correct result. But in the reported case, VersionConflict is raised not because the to-be-installed package conflicts, but one of its dependencies do. So the second get_distribution("{package}") still fails, resulting in a strange error.

I am not sure how this should be fixed. Maybe pkg_resources.get_distribution() should grow a flag so it can be used without checking conflicts in dependencies, or maybe should not use this function to get an installed distribution in the first place. Or maybe pip should uninstall that conflicting dependency (it will if this code block works anyway).

@1313e A workaround is to run python setup.py egg_info before pip install -e. This makes pip choose the 1. code path above, avoiding the bug.

[cjerdonek]

Great! And why was this limited to Travis? I'm assuming that was a red herring. Were you able to reproduce locally after all?

[cjerdonek]

Re: your proposed options, sometimes adding a flag to an existing function adds undesired complexity because of branching and making the call sites relying on that flag harder to find. Would it make sense to add a function to pkg_resources that is parallel to and shares code with get_distribution()? That way logic wouldn't be getting duplicated.

[uranusjr]

Yes I was! I wasn’t able to reproduce before because my test case was either too clean or not clean enough (a prebuilt egg-info resolves the problem). Travis has an environment right in the sweet spot, and local reproduction is easy once I know where that spot is :p

[1313e]

@uranusjr Wow, that turned out to be a lot harder and more complex than I thought.
I will use that workaround for now.

[Arusekk]

What is the status on this?

[pradyunsg]

@uranusjr This is a fairly big deviation from the resolvelib work for you, but any idea what we could change in pip to make this less likely for end users?

[1313e]

Does this not have to do with the dependency resolver issues discussed in #988?

[uranusjr]

Does this not have to do with the dependency resolver issues discussed in #988?

Yes, eventually the resolver work would make this whole block obsolete.

---

Any idea what we could change in pip to make this less likely for end users?

If we switch to importlib.metadata (#7413) it would stop being a problem for Python 3 users. The code here is not wrong in general, just using the wrong function (pkg_resources.get_distribution()).

[pradyunsg]

OK great! Given that our broader chunks of work would end up fixing this, I'll stop spending time trying to understand why we're having this issue; and instead spend it on progressing on those tasks. :)