With Enum34 installed on python3.6+, pip picks up enum34 instead of the stdlib enum package

[abadger]

Environment

• pip version: pip 20.1
• Python version: Python 3.6.10 [This was reported to me against python3.6 so I tested and fixed it there but it should affect later versions of python as well]
• OS: Fedora 31

Description

When the enum34 package is installed on a system, pip will no longer install a project that uses pyproject.toml instead of setup.py. Instead, it tracebacks inside of the stdlib because pip is finding the enum34 package's version of enum.py instead of the stdlib version. This happens in the re-invocation of pip from pip/_internal/commands/install.pyhttps://github.com/abadger/pip/blob/master/src/pip/_internal/build_env.py#L171 although I think the bug should be fixed in __main__.py instead.

(edited: Put the correct file into the description)

The reason is that __main__.py has code which changes sys.path in case we are running pip from a wheel. That code appears to be changing sys.path in certain circumstances when we aren't running from a wheel by mistake. So that code is where the bug lies, not in install. It may be correct to change install.py as well but I am leaning towards not doing that... It looks like the code in install is making sure that it reinvokes itself by specifying the actual path to pip which it was invoked with. If we tried to use something different like python -m there, then we wouldn't be assured of executing the code from the same install of pip.

I'll add an easy reproducer using pip's repository in the reproducer section.

Fixed by #8213

Expected behavior

Even if enum34 is installed, pip should still be able to install other packages with a pyproject.toml file.

How to Reproduce

1. python3.6 -m pip install --user enum34
2. python3.6 -m pip install --user git+git://github.com/pypa/pip
3. A traceback occurs:

Output

[pts/16@peru /srv/git/python-pip/pip/news]$ python3.6 -m pip install --user enum34
/usr/lib64/python3.6/enum.py
Requirement already satisfied: enum34 in /home/badger/.local/lib/python3.6/site-packages (1.1.10)
Could not build wheels for enum34, since package 'wheel' is not installed.
[pts/16@peru /srv/git/python-pip/pip/news]$ python3.6 -m pip --version
/usr/lib64/python3.6/enum.py
pip 20.1 from /home/badger/.local/lib/python3.6/site-packages/pip (python 3.6)
[pts/16@peru /srv/git/python-pip/pip/news]$ python3.6 --version *[fix-pip-to-work-in-the-face-of-enum34]  (08:47:21)
Python 3.6.10
[pts/16@peru /srv/git/python-pip/pip/news]$ python3.6 -m pip install --user git+git://github.com/pypa/pip
Collecting git+git://github.com/pypa/pip
  Cloning git://github.com/pypa/pip to /var/tmp/pip-req-build-edpkq5kv
  Running command git clone -q git://github.com/pypa/pip /var/tmp/pip-req-build-edpkq5kv
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3.6 /home/badger/.local/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /var/tmp/pip-build-env-aap_1xn0/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel
       cwd: None
  Complete output (14 lines):
  Traceback (most recent call last):
    File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
      "__main__", mod_spec)
    File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "/home/badger/.local/lib/python3.6/site-packages/pip/__main__.py", line 23, in <module>
      from pip._internal.cli.main import main as _main  # isort:skip # noqa
    File "/home/badger/.local/lib/python3.6/site-packages/pip/_internal/cli/main.py", line 5, in <module>
      import locale
    File "/usr/lib64/python3.6/locale.py", line 16, in <module>
      import re
    File "/usr/lib64/python3.6/re.py", line 142, in <module>
      class RegexFlag(enum.IntFlag):
  AttributeError: module 'enum' has no attribute 'IntFlag'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3.6 /home/badger/.local/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /var/tmp/pip-build-env-aap_1xn0/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel Check the logs for full command output.

[pfmoore]

This happens in the re-invocation of pip from pip/_internal/commands/install.py

Can you provide a link to the relevant line of code? I can't find what you're referring to here 🙁

Also, I just tried this and it worked fine for me:

[root@696e6f345c66 /]# python3.6 -m pip install --user enum34
WARNING: Running pip install with root privileges is generally not a good idea. Try `python3.6 -m pip install --user` instead.
Collecting enum34
  Downloading https://files.pythonhosted.org/packages/63/f6/ccb1c83687756aeabbf3ca0f213508fcfb03883ff200d201b3a4c60cedcc/enum34-1.1.10-py3-none-any.whl
Installing collected packages: enum34
Successfully installed enum34-1.1.10
WARNING: You are using pip version 19.1.1, however version 20.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[root@696e6f345c66 /]# python3.6 -m pip install --user git+git://github.com/pypa/pip
WARNING: Running pip install with root privileges is generally not a good idea. Try `python3.6 -m pip install --user` instead.
Collecting git+git://github.com/pypa/pip
  Cloning git://github.com/pypa/pip to /tmp/pip-req-build-st28fks3
  Running command git clone -q git://github.com/pypa/pip /tmp/pip-req-build-st28fks3
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: pip
  Building wheel for pip (PEP 517) ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-fbj8jirm/wheels/1f/11/b0/5bd125f64cf9eedf300124c783996f30957e0d1b4fac0c38d3
Successfully built pip
Installing collected packages: pip
Successfully installed pip-20.2.dev0

[abadger]

@pfmoore It's this line here: https://github.com/abadger/pip/blob/master/src/pip/_internal/build_env.py#L171

/me looks through your output to see if he can spot what is different between our two environments.

[abadger]

pip_location is defined at the top of the file like this: from pip import __file__ as pip_location

[abadger]

Oh! I think I might know why you couldn't reproduce. Where is your initial pip located? My initial pip is also in ~/.local. That would come into play because the line in build_env.py is going to putresult in a directory relative to the pip installation you are running into sys.path. So if you were running the initial pip from /usr/lib/[...]/site-packages but installed enum34 into ~/.local/[...]/site-packages, then you wouldn't trigger the issue where enum from enum34 is used in preference to the one from the stdlib.

/me goes forth to verify this is the difference now that he's posted the theory here.

EDITED: My language was unclear iniitally. The code in build_env re-invokes pip as sys.executable /path/to/the/pip/install/directory/. This results in a directory relative to the pip installation to end up in the re-invoked pip's sys.path. However, the code in pip/__main__.py is what actually puts that directory into sys.path.

[abadger]

Yep, that's what it was. So if you want to reproduce from where you said it worked for you, you can simply run the last pip command again. Since you just installed pip from git into the user home dir, the second time the command runs, it will use that pip and then fail.

If you want to start over, here are all the steps I took with a container to reproduce (Leaving off output but adding comments)

# Host computer
podman run -it fedora /bin/bash
# Now inside the container
# Running ensurepip with --user puts pip into the same site-packages as enum34
# will be installed into
python3.8 -m ensurepip --user
python3.8 -m pip install --user enum34
python3.8 -m pip install --user git+git://github.com/pypa/pip
# Traceback from the initial bug report here

[abadger]

Oops, and I see why you couldn't find the line I was talking about earlier too... I posted the wrong file at the top of the issue. Sorry. I was greping the code for the command line strings to see if I could find any other occurrences that would have the same problem and I cut and pasted the wrong filename into the issue report.

[felixfontein]

In python-poetry/poetry#1122 this happened with Python 3.7.3.

[pfmoore]

So this is related to running pip for the isolated build environment. That makes sense, and was what I initially assumed you might be thinking about, but your reference to the wrong file put me off track. Thanks for clarifying.

So I think there probably is something to address here, but I'm not sure your proposed fix is the right approach. I still think that running pip as python /path/to/pip isn't something we should support - and if that means we need to consider another way of running pip in build_env.py then let's look at that.

[abadger]

. Yeah, that's fair. So there's several things I am thinking....

• From the comment in the source code of __main__.py I interpret that the current code in __main__ is overzealous in what it adds. If you don't think the scope of that code should be decreased, what should the comment actually say? What use case in addition to a pip wheel is it meant to address?
• It seems like there should be a different way to address this particular issue inside of build_env. The strand that has to be pulled apart for any other solution to work, though, is that it needs to load the correct library for pip without affecting the other libraries that might be pulled in. Python has all sorts of ways to invoke pip but half of them will end up affecting sys.path in a global manner. I think you could separate out the pip library that you want to use (turn it into a wheel or copy the directory structure to its own directory or etc.) Then place that onto PYTHONPATH before re-invoking pip using sys.executable -m That way only the library that you separated out would override everything else on the PYTHONPATH.

[abadger]

I've figured out another way to change build_env.py that looks like it works.... but I'm afraid it might be an undocumented implementation detail of the way python works so I'm not sure if it's safe to rely on. Instead of executing the directory where pip lives, execute pip's __main__.py file instead. Executing the directory will end up setting __package__ to the empty string. Executing the __main__.py file will set __package__ to None. __main__.py only adds to sys.path if __package__ is the empty string so it will work for this as well (Probably needs to have a comment if you go this route, though, so that everyone knows that the condition needs to be True for empty string and False for None).

If you want to go that route, it would look something like this:

diff --git a/src/pip/__main__.py b/src/pip/__main__.py
index a529758b..2e9ff7e7 100644
--- a/src/pip/__main__.py
+++ b/src/pip/__main__.py
@@ -12,16 +12,16 @@ if sys.path[0] in ('', os.getcwd()):
 
 # If we are running from a wheel, add the wheel to sys.path
 # This allows the usage python pip-*.whl/pip install pip-*.whl
+# Do not enter this codepath if __package__ is None as that
+# means that pip/__main__.py was invoked directly, which is what
+# build_env.py does (issue #8214)
 if __package__ == '':
     # __file__ is pip-*.whl/pip/__main__.py
     # first dirname call strips of '/__main__.py', second strips off '/pip'
 
diff --git a/src/pip/_internal/build_env.py b/src/pip/_internal/build_env.py
index b8f005f5..77e192f2 100644
--- a/src/pip/_internal/build_env.py
+++ b/src/pip/_internal/build_env.py
@@ -167,8 +167,9 @@ class BuildEnvironment(object):
         prefix.setup = True
         if not requirements:
             return
+        command = os.path.join(os.path.dirname(pip_location), '__main__.py')
         args = [
-            sys.executable, os.path.dirname(pip_location), 'install',
+            sys.executable, command, 'install',
             '--ignore-installed', '--no-user', '--prefix', prefix.path,
             '--no-warn-script-location',
         ]  # type: List[str]

[abadger]

Documentation on __package__ that I can find:

• https://docs.python.org/3/reference/import.html
  • __package__ must be a string.
  • __package__ will be the empty string when the module is not a package.
• https://www.python.org/dev/peps/pep-0366/#id11
  • Documents instances in which __package__ may be set to None.
  • In particular, I believe this applies here: "When the main module is specified by its filename, then the __package__ attribute will be set to None."

If you want to go this route, perhaps we should ping ncoghlan at that time to see if he considers the None/empty string split to be something that we can depend upon here.