pip install -e crashes on Python 3.8 when 'typing' is installed from PyPI because it runs a sub-command in such a way that site-packages comes before lib/python3.8

[glyph]

Environment

• pip version: 20.1, 20.1.1
• Python version: 3.8.2
• OS: macOS, linux

Description
pip install -e fails with a traceback if the typing module is installed from PyPI.

Expected behavior
It should work, like it used to.

How to Reproduce

python3.8 -m venv foo
./foo/bin/pip install typing
./foo/bin/pip install -e ~/src/twisted

Output

  ERROR: Command errored out with exit status 1:
   command: /Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/bin/python /Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/z9/w23lcw3x70z1fqpcwjjbvkvc0000gp/T/pip-build-env-f7o2d80a/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i http://127.0.0.1:3141/root/plus/+simple/ -- 'setuptools>=40.8.0' wheel
       cwd: None
  Complete output (44 lines):
  Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code
      exec(code, run_globals)
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/__main__.py", line 26, in <module>
      sys.exit(_main())
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 73, in main
      command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_internal/commands/__init__.py", line 104, in create_command
      module = importlib.import_module(module_path)
    File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
    File "<frozen importlib._bootstrap>", line 991, in _find_and_load
    File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 783, in exec_module
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 24, in <module>
      from pip._internal.cli.req_command import RequirementCommand, with_cleanup
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 16, in <module>
      from pip._internal.index.package_finder import PackageFinder
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 21, in <module>
      from pip._internal.index.collector import parse_links
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 14, in <module>
      from pip._vendor import html5lib, requests
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_vendor/requests/__init__.py", line 114, in <module>
      from . import utils
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_vendor/requests/utils.py", line 25, in <module>
      from . import certs
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_vendor/requests/certs.py", line 15, in <module>
      from pip._vendor.certifi import where
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_vendor/certifi/__init__.py", line 1, in <module>
      from .core import contents, where
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip/_vendor/certifi/core.py", line 12, in <module>
      from importlib.resources import read_text
    File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/resources.py", line 11, in <module>
      from typing import Iterable, Iterator, Optional, Set, Union   # noqa: F401
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/typing.py", line 1357, in <module>
      class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
    File "/Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/typing.py", line 1005, in __new__
      self._abc_registry = extra._abc_registry
  AttributeError: type object 'Callable' has no attribute '_abc_registry'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/bin/python /Users/glyph/.virtualenvs/tmp-7b0fea29ffcfc56/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/z9/w23lcw3x70z1fqpcwjjbvkvc0000gp/T/pip-build-env-f7o2d80a/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i http://127.0.0.1:3141/root/plus/+simple/ -- 'setuptools>=40.8.0' wheel Check the logs for full command output.

[glyph]

I think this might just be a duplicate of #8214 ?

[pfmoore]

It is indeed.

It should work, like it used to.

When you say "it used to", when was the last time it worked? My feeling is that it would not have worked since we introduced build isolation. Does that match your experience?

[glyph]

It worked in every version prior to 20.1, so 20.0.2.

[pfmoore]

Thanks, that's useful to know.

[pradyunsg]

Gah, this is likely a regression, since nearly all of pip's code avoids importing typing as much as possible. I'm curious how this slipped through, but we can look into this separately.

Looks like another bugfix release is needed. :(

[pfmoore]

It looks like #5841 introduced this, triggering the issue with how we set sys.path in __main__.

[pfmoore]

I'm now inclined to accept #8213 as a fix for this. It's a bit of a hack, but so's all of the code in this area, and now that we understand what's going on, and have confirmation that it's affecting things more widely than the original report suggested, it seems better to apply that fix and worry later about cleaning up build_env a bit better.

[uranusjr]

I think it’d be better to just revert #5841 than accepting #8213. They would both break the usage #5841 tried to fix, but at least reversion is unlikely to introduce more problems.

[pfmoore]

Ah sorry, you're right. I was too quick in my comment - indeed #8213 breaks #5841, I was reading the conditional wrongly.

I still think we need to understand the problem better, and I don't think the issue is urgent enough to need another bugfix release (all of the reproducers I've seen involve installing a backport module in a Python release that doesn't need it - so "don't do that" seems like a sufficient workaround for the immediate term).

One thing that bothers me - although I haven't tested anything myself - is that this is reported as working in 20.0.2, which is odd as #5841 was included from 19.0, and the sys.path code has been there since forever. So what changed after 20.0.2 that affected this?

[pradyunsg]

I don't think the issue is urgent enough to need another bugfix release (all of the reproducers I've seen involve installing a backport module in a Python release that doesn't need it - so "don't do that" seems like a sufficient workaround for the immediate term).

Yea, that makes sense to me! :)

[tony]

Possibly related

mypy-extensions https://pypi.org/project/typing-extensions/#history

3.7.4.2 on Apr 2nd, 2020

python/typing@f254d69#diff-85eeb059061bb009c803d065dce510ec

https://github.com/python/typing/commits/3.7.4.1/typing_extensions/setup.py