AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?

[aorumbayev]

Describe the problem

The following repo was specifically created to outline a reproducible environment with the bug https://github.com/aorumbayev/pipx_bug.

The root cause is incompatibility with python 3.12. And this particular line is the root cause. I will also open this issue on pipx repo, so far i mentioned it on copier where copier maintainers forwarded me to this repo.

To Reproduce

Refer to readme on https://github.com/aorumbayev/pipx_bug

Logs

Traceback (most recent call last):
  File "/usr/local/py-utils/bin/pipx-bug-cli", line 5, in <module>
    from pipx_bug.cli import hello
  File "/usr/local/py-utils/venvs/pipx-bug/lib/python3.12/site-packages/pipx_bug/cli.py", line 2, in <module>
    from copier.main import Worker
  File "/usr/local/py-utils/venvs/pipx-bug/lib/python3.12/site-packages/copier/__init__.py", line 6, in <module>
    from .main import *  # noqa: F401,F403
    ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/py-utils/venvs/pipx-bug/lib/python3.12/site-packages/copier/main.py", line 33, in <module>
    from plumbum.cli.terminal import ask
  File "/usr/local/py-utils/venvs/pipx-bug/lib/python3.12/site-packages/plumbum/cli/__init__.py", line 1, in <module>
    from .application import Application
  File "/usr/local/py-utils/venvs/pipx-bug/lib/python3.12/site-packages/plumbum/cli/application.py", line 9, in <module>
    from plumbum.cli.i18n import get_translation_for
  File "/usr/local/py-utils/venvs/pipx-bug/lib/python3.12/site-packages/plumbum/cli/i18n.py", line 28, in <module>
    import pkg_resources
  File "/usr/local/py-utils/shared/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2191, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?

Expected behavior

plumbum works when used as a transitive dependency in packages installed via pipx

Operating system

Linux

Operating system distribution and version

ubuntu 20

Plumbum version

1.8.2

Python version

3.12.2

Installation method

plumbum is introduced as a transient dependency of copier. https://github.com/copier-org/copier/blob/85311ec658ea7162673489f70e7ec5a8d6b5367b/poetry.lock#L862

[henryiii]

import pkg_resources should not be broken if setuptools is installed. You have an old version somehow, which probably means setuptools was capped in some package to a version that doesn't support Python 3.12. What version of setuptools is in your pipx environment?

[henryiii]

Ahh, this is getting the surrounding environment. What is your PYTHONPATH set to? Setting that is one way to break pipx, I think.

[aorumbayev]

@henryiii, could you please confirm if you've had a chance to execute the public repository that demonstrates the bug in a GitHub Codespace (specifically created to allow folks to easily reproduce the bug)?

Here's the sys.path output from the pipx environment for the problematic package:

Python 3.12.2 (main, Apr 23 2024, 13:31:25) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.path)
['', '/home/codespace/.pyenv/versions/3.12.2/lib/python312.zip', '/home/codespace/.pyenv/versions/3.12.2/lib/python3.12', '/home/codespace/.pyenv/versions/3.12.2/lib/python3.12/lib-dynload', '/usr/local/py-utils/venvs/algokit/lib/python3.12/site-packages', '/usr/local/py-utils/shared/lib/python3.10/site-packages']

Setuptools is already installed in the virtual environment using Python 3.12.2:

Requirement already satisfied: setuptools in /home/codespace/.local/lib/python3.12/site-packages (69.2.0)

It appears that Python 3.10 directories are being added to sys.path, likely due to pipx. There have been recommendations in similar pipx issues to perform a full reinstall of pipx to reset the shared virtual environment (created with Python 3.10). This suggests a possible challenge with packages requiring Python >=3.12 in GitHub Codespaces, as users might need to completely uninstall the preinstalled pipx and reinstall it with Python 3.12.

Additionally, using pipx reinstall-all does not seem to update the shared virtual environment if it was initially set up with Python 3.10—pipx retains it. This behavior is unexpected and complicates environment management a lot :/

[henryiii]

'/usr/local/py-utils/venvs/algokit/lib/python3.12/site-packages', '/usr/local/py-utils/shared/lib/python3.10/site-packages' is a broken sys.path. There's no guarantee any package will work when you have two versions of site-packages for different Python's in your path. Pipx won't add another Python to your sys.path, and it will place directories at the top of the PATH, so that's already present on your system. You happen to hit it here because we don't require setuptools, but check to see if it's present. It is present, but in your Python 3.10 site packages, so it's mixing with the Python 3.12 environment and the whole thing breaks down.

If you fix your sys.path, then this problem goes away. You should not have Python 3.10 in your sys.path if you are using Python 3.12.

There are a few things that could be done to alleviate the problem (basically just push it off on some other package) from our side, though:

• Check for AttributeError in addition to ImportError when skipping. A little weird though.
• Require setuptools to be installed.
• Switch to importlib_resources/importlib.resources.

Of those, the most effort is the third one, but my preferred solution.

[henryiii]

I really don't see where that python3.10 is sneaking in. Everything seems fine in the container, pyvenv.cfg looks correct, no 3.10's in the environment. Maybe there's an issue using pyenv in the container that's already configured for 3.10? I don't really trust pyenv or tools to support it very well (due to the messy shim system). If you can use a better image in devcontainer.json, or use a different mechanism to install Python (PDM and Hatch both can install Python, unlike Poetry), this might be resolved.

[henryiii]

Same issue with pdm:

pipx install pdm
pdm python install cpython@3.12.2
pipx install . --force --python /home/codespace/.local/share/pdm/python/cpython@3.12.2/bin/python3
pipx-bug-cli

Though it's many times faster since these tools (pdm, hatch, rye) don't build Python from source like pyenv does.

I notice the Codespaces is detecting the Poetry file and doing some setup, not sure if that affects anything.

[henryiii]

Also, POSIX has a "C" locale which we should also handle. We can easily fix that, which solves this if the locale is not set to non-English.

[aorumbayev]

@henryiii thanks for the investigation so far! Curious if #680 fixed the issue? Will try testing it out in the codespace as well

[henryiii]

#681 should have fixed it. It’s technically a broken environment, so just a patch to help reduce the issue was fine.

Will finish up #680 after PyCon.

[aorumbayev]

@henryiii colleagues have reported that they have hit the issue again, still happening when using latest copier that relies on plumbum (they don't pin the dependency) so it should be picking up the fix in #681 as it was merged already. I suppose its worth reopening the issue on copier as well.