Support for Python(s) (3.10+?) without distutils.

[MatthewRalston]

Be sure to check the existing issues (both open and closed!), and make sure you are running the latest version of Pipenv.

Check the diagnose documentation for common issues before posting! We may close your issue if it is very similar to one of them. Please be considerate, or be on your way.

Make sure to mention your debugging experience if the documented solution failed.

Issue description

I believe there is a compatibility issue for pipenv with Python 3.12, I think as far back as 3.11-10 the buildchain was undergoing some upgrades, including that python no longer supports distutils.

Expected result

I am encountering some issues in my usage of pipenv. I am using a too-recent buildchain because I am struggling to find a way to support current python, given some libraries individual reliance on the deprecated distutils dependency in the buildchain. I would like to find a way to support multiple versions simultaneously, and sometimes I get locked up in dependency hell because something essential is in flux, or that my own repository health seems like a spookytime echo of my own sometimes.

Actual result

>pipenv lock
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!
⠏ Locking...False

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/pipenv/resolver.py", line 675, in <module>
    main()
  File "/usr/local/lib/python3.12/site-packages/pipenv/resolver.py", line 661, in main
    _main(
  File "/usr/local/lib/python3.12/site-packages/pipenv/resolver.py", line 645, in _main
    resolve_packages(
  File "/usr/local/lib/python3.12/site-packages/pipenv/resolver.py", line 612, in resolve_packages
    results, resolver = resolve(
                        ^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/resolver.py", line 592, in resolve
    return resolve_deps(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 918, in resolve_deps
    results, hashes, internal_resolver = actually_resolve_deps(
                                         ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 691, in actually_resolve_deps
    resolver.resolve()
  File "/usr/local/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 446, in resolve
    results = resolver.resolve(constraints, check_supported_wheels=False)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_vendor/resolvelib/resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
    return bool(self._sequence)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
    return any(self)
           ^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
    candidate = func()
                ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/factory.py", line 182, in _make_candidate_from_link
    base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/factory.py", line 228, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
                                       ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/candidates.py", line 297, in __init__
    super().__init__(
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/candidates.py", line 157, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/candidates.py", line 226, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/candidates.py", line 308, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_internal/operations/prepare.py", line 525, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_internal/operations/prepare.py", line 640, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_internal/operations/prepare.py", line 71, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_internal/distributions/sdist.py", line 54, in prepare_distribution_metadata
    self._install_build_reqs(finder)
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_internal/distributions/sdist.py", line 124, in _install_build_reqs
    build_reqs = self._get_build_requires_wheel()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_internal/distributions/sdist.py", line 101, in _get_build_requires_wheel
    return backend.get_requires_for_build_wheel()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_internal/utils/misc.py", line 751, in get_requires_for_build_wheel
    return super().get_requires_for_build_wheel(config_settings=cs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_vendor/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_vendor/pyproject_hooks/_impl.py", line 321, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pipenv.patched.pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/pipenv/patched/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
    obj = import_module(mod_path)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File /tmp/pip-build-env-qr5b0i_2/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 10, in <module>
    import distutils.core
ModuleNotFoundError: No module named 'distutils'

Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/cli/options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/vendor/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/cli/command.py", line 340, in lock
    do_lock(
  File "/usr/local/lib/python3.12/site-packages/pipenv/routines/lock.py", line 65, in do_lock
    venv_resolve_deps(
  File "/usr/local/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 859, in venv_resolve_deps
    c = resolve(cmd, st, project=project)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 728, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

Be me

Download Python 3.4+

Need to support modern modules for latest features and buildchain 'fidelity'
Install Python 3.12.1
Try to get away from pyenv/pyenv ( >__>)

keep trying...

help

keep trying

(me drowning):
....oO...0OO....0.O.O..0.0..0.0..0..0......0.00.....0.0...0...OO.O....OO..00.0..0..0..0......0......0.0..oo...o.o..o.o...o.o..o,.............................................,,,,,. ..... ,,,,,,,,,,,,,,,,,,,,,,,,,,,,, ,,,,,,,........ . . .

think oh wait "pipenv is pypa"

crashes on Pythons 3.7 3.8 3.9 3.10 + on my system.

Not their fault, 99% user error probability.

still think "its pypa"
go to this issue tracker
go to virtualenv

ahaha
its not pypa specifically.

"but who inherited it"?

virtualenv i guess.

buildchain woes.

pipenv tastes sweet. But "it's actually a dependency bug" was said in the threads I read about the distutil dependency. I probably read the wrong ones. Okay?

except vanilla commands (pipenv lock, pipenv install) produce RuntimeErrors and ModuleNotFoundError errors regarding what is sometimes referred to as the "buildchain" and virtualenv/distutils/setuptools/{install_wizardry}

be told my first report is user issue on #6124 using the Ubuntu 20.04 ppa repository 'pipenv'.

I started there cause that was the first issue to debug.

The buildchain issue I'm showing here has nothing to do with pipenv per-se, however... it relates to the entire buildchain and pip (i.e. pypa) architechture for Python packages.

Okay, that's what I think about this error.

Thank you PyPA and contributors.

#6124

---

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

If you're on macOS, run the following:

$ pipenv --support | pbcopy

If you're on Windows, run the following:

> pipenv --support | clip

If you're on Linux, run the following:

$ pipenv --support | xclip

Contents of Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
numpy = ">=1.21.2"
pyyaml = ">=6.0.1"
jsonschema = ">=4.17.3"
psutil = ">=4.2.0"
cython = ">=3.0.8"
biopython = ">=1.81"
scipy = ">=1.7.3"
scikit-learn = "==1.0.2"
matplotlib = ">=3.5.3"

[dev-packages]

[requires]
python_version = "3.12"
python_full_version = "3.12.2"

[matteius]

@MatthewRalston I am having a slightly hard time following this, can you verify what version of pipenv is installed?

[matteius]

Possibly this is related: #5674 (comment)
Or this: #5674 (comment)

[MatthewRalston]

@MatthewRalston I am having a slightly hard time following this, can you verify what version of pipenv is installed?

It's whatever the latest and greatest is on the Ubuntu ppa. It's obviously not related to that specific version, it's an issue in the 3.10+ build chain where distutila is no longer available.

Does that clarify?

[matteius]

@MatthewRalston well you need to check, because depending on the version of ubuntu you are on, that is 11.9.0-1.1 from 2019-10-18 which is ancient. pipenv --version also, try uninstalling that and installing from pypi. We support pypi installs.

[MatthewRalston]

That's not the problem IMO.

this seems to be a non-issue for PyPA (not you, per se. And thanks anyways) that Pythons setuptools/distutil chain has changed, and collides with pipenv when Python version is greater than 3.10. this is a PyPA and toolchain issue and not user error.

I'm closing this since there's no traction.

[matteius]

@MatthewRalston well its not a matter of opinion what version of pipenv you are running, if its 11.9.x it won't work, if its 2022.12.1 there might be an issue here that we can help you resolve. I'm not as familiar with the distutils stuff but it isn't referenced in our code base outside of pip vendoring and some old test artifacts. I just spent last night forming a branch that converts pipenv off pkg_resources -- you could try installing that branch from github (check the PRs) and see if it resolves the issue too.

[MatthewRalston]

Okay. Installed from PyPA directly instead of GitHub release or Ubuntu 20.04 ppa.

Can confirm that the PyPA pipenv 2023.12.1 does not produce the distutil error on Python 3.12. thank you!

[melyux]

I'm on Python 3.12.3 and pipenv 2023.12.1, still getting ModuleNotFoundError: No module named 'distutils' during pipenv install.

[matteius]

@melyux please try the main branch -- we are still working towards the next release.