ImportError: cannot import name 'SourceDistribution' from 'pip._internal.distributions.source'

[peterjc]

Environment

CircleCI using Conda, eg https://circleci.com/gh/peterjc/thapbi-pict/1073

• pip version: pip-19.2.3 (from conda) updated to 19.3 (via pip install --upgrade pip setuptools)
• Python version: python-3.7.3
• OS: Linux

I have made only minor changes to my code, but today the CircleCI builds broke during the build/install stage of my Python tool - multiple branches affected. The timing fits the release of pip 19.3 on PyPI.

Description

It appears a recent release of pip moved some internal files about, my guess is #6830.

Expected behavior

How to Reproduce

Full output on https://circleci.com/gh/peterjc/thapbi-pict/1073 which is running https://github.com/peterjc/thapbi-pict/blob/9fcec2da60e6e6ae5cf7ee6ad4b53dcc3a40cfe7/.circleci/config.yml#L56

...
python setup.py sdist --formats=gztar
python setup.py bdist_wheel
pip install dist/thapbi_pict-*.whl

Output

Traceback (most recent call last):
  File "/opt/conda/bin/pip", line 7, in <module>
    from pip._internal import main
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/__init__.py", line 40, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/cli/autocompletion.py", line 8, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/cli/main_parser.py", line 11, in <module>
    from pip._internal.commands import (
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/commands/__init__.py", line 9, in <module>
    from pip._internal.commands.download import DownloadCommand
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/commands/download.py", line 10, in <module>
    from pip._internal.operations.prepare import RequirementPreparer
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 9, in <module>
    from pip._internal.distributions import (
  File "/opt/conda/lib/python3.7/site-packages/pip/_internal/distributions/__init__.py", line 1, in <module>
    from pip._internal.distributions.source import SourceDistribution
ImportError: cannot import name 'SourceDistribution' from 'pip._internal.distributions.source' (/opt/conda/lib/python3.7/site-packages/pip/_internal/distributions/source/__init__.py)
Exited with code 1

[pfmoore]

It would be really helpful to know what in the CircleCI (or conda?) setup is using those internal APIs - the reorganisation shouldn't have affected any external code. The main issue we know of so far is the reorganisation of main, which impacts wrapper scripts, but this one looks a bit different. Any insight you can give would be very useful, thanks.

[peterjc]

Confirmed that removing pip install --upgrade pip setuptools fixed things,

peterjc/thapbi-pict@4060404
https://circleci.com/gh/peterjc/thapbi-pict/1074

i.e. Something broke in how I updated from pip 19.2.3 to 19.3

I am not intentionally using the CircleCI provided Python, rather the Conda provided Python - is there value in switching from pip ... to python -m pip ... just to remove the possibility of this being due to mixing Pythons?

[peterjc]

Working on a branch now, added a little more logging and reverted back to the broken version:

https://circleci.com/gh/peterjc/thapbi-pict/1076

This confirms the traceback is trigged by pip install dist/thapbi_pict-*.whl (wildcard to match the single freshly built wheel).

[peterjc]

OK, using python -m pip ... rather than pip ... does not help:

https://circleci.com/gh/peterjc/thapbi-pict/1077
peterjc/thapbi-pict@c168fbb

[pfmoore]

For what it's worth, I've just confirmed that from pip._internal import main works on a simple upgrade of pip (on Windows, so not really a good comparison, but anyway...). So I suspect there's something mismatched in your environment, although I can't see from the traceback what it is.

[peterjc]

CircleCI has the capability to re-run a job with SSH access to the VM (not that I've ever used it), so in theory this could be used for debugging, but I don't really know where to start.

I have a work around for now, and at some point conda-forge will have pip 19.3 packaged too, so unless other people see the same exception, it seems a low priority?

[pfmoore]

OK, cool. If you have a workaround then I'm fine leaving it there for now. If others add comments reporting it's happening elsewhere, we can do some more digging then - maybe they will have extra information that will help.

[maulin-amwater]

I am facing a similar issue on macOS, python: 3.6.9. pip command on my terminal doesn't work anymore. Any suggestions?

[maulin-amwater]

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py

This helped for now

[nshmura]

I have same problem on CircleCI since 1 hours ago.
Version changing of docker image from python:3.7 to python:3.7.4 fixes this issue. (just workaround) . like this:

version: 2
...
jobs:
  build:
    docker:
      - image: python:3.7.4
...

[bgkelly]

Are any of you guys using caching? I think it's possible some pip internals changed and when you restore the cache of the old python version pip breaks. Suggest you invalidate your circle CI caches by changing the keys and see what happens ... (worked for me!)