Repository navigation
ModuleNotFoundError: No module named 'distutils' in latest release #993
Description
Activity
Same issue in our pipelines: https://github.com/microsoft/torchgeo/actions/runs/9356814161/job/25755818665?pr=2103
Same issue here...
distutilswas removed in Python 3.12Would also be good to get the source build working in Python 3.12+ even if macOS arm64 wheels are available. May be as simple as removing the pin to older setuptools versions?
MacOS ARM wheels are now available, but as mentioned the sdist should probably still be fixed
Related to this issue, the python:alpine image includes the updated
setuptoolswhich resolves the missingdistutilsmodule but installation ofopencv-pythonstill fails.For context, trying to build my own package which includes opencv-python as a dependency, but I am having issues installing opencv-python using pip in alpine 3.20 (python3.12).
Steps to reproduce using simplified dockerfile and no source code
FROM python:alpine RUN pip3 install opencv-python
then:
$ docker build -t opencv-python . [+] Building 10.8s (6/6) FINISHED docker:default => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 131B 0.0s => [internal] load metadata for docker.io/library/python:alpine 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [1/3] FROM docker.io/library/python:alpine 0.0s => CACHED [2/3] RUN pip3 install --upgrade setuptools pip 0.0s => ERROR [3/3] RUN pip3 install opencv-python 10.8s ------ > [3/3] RUN pip3 install opencv-python: 0.669 Collecting opencv-python 0.939 Downloading opencv-python-4.10.0.84.tar.gz (95.1 MB) 3.129 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 95.1/95.1 MB 27.7 MB/s eta 0:00:00 5.320 Installing build dependencies: started 10.43 Installing build dependencies: finished with status 'done' 10.44 Getting requirements to build wheel: started 10.51 Getting requirements to build wheel: finished with status 'error' 10.51 error: subprocess-exited-with-error 10.51 10.51 × Getting requirements to build wheel did not run successfully. 10.51 │ exit code: 1 10.51 ╰─> [33 lines of output] 10.51 Traceback (most recent call last): 10.51 File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module> 10.51 main() 10.51 File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main 10.51 json_out['return_val'] = hook(**hook_input['kwargs']) 10.51 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 10.51 File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 112, in get_requires_for_build_wheel 10.51 backend = _build_backend() 10.51 ^^^^^^^^^^^^^^^^ 10.51 File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend 10.51 obj = import_module(mod_path) 10.51 ^^^^^^^^^^^^^^^^^^^^^^^ 10.51 File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module 10.51 return _bootstrap._gcd_import(name[level:], package, level) 10.51 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 10.51 File "<frozen importlib._bootstrap>", line 1387, in _gcd_import 10.51 File "<frozen importlib._bootstrap>", line 1360, in _find_and_load 10.51 File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked 10.51 File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed 10.51 File "<frozen importlib._bootstrap>", line 1387, in _gcd_import 10.51 File "<frozen importlib._bootstrap>", line 1360, in _find_and_load 10.51 File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked 10.51 File "<frozen importlib._bootstrap>", line 935, in _load_unlocked 10.51 File "<frozen importlib._bootstrap_external>", line 995, in exec_module 10.51 File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed 10.51 File "/tmp/pip-build-env-rzx3tyxe/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module> 10.51 import setuptools.version 10.51 File "/tmp/pip-build-env-rzx3tyxe/overlay/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module> 10.51 import pkg_resources 10.51 File "/tmp/pip-build-env-rzx3tyxe/overlay/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2172, in <module> 10.51 register_finder(pkgutil.ImpImporter, find_on_path) 10.51 ^^^^^^^^^^^^^^^^^^^ 10.51 AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'? 10.51 [end of output] 10.51 10.51 note: This error originates from a subprocess, and is likely not a problem with pip. 10.51 error: subprocess-exited-with-error 10.51 10.51 × Getting requirements to build wheel did not run successfully. 10.51 │ exit code: 1 10.51 ╰─> See above for output. 10.51 10.51 note: This error originates from a subprocess, and is likely not a problem with pip. ------ Dockerfile:4 -------------------- 2 | 3 | RUN pip3 install --upgrade setuptools pip 4 | >>> RUN pip3 install opencv-python 5 | 6 | -------------------- ERROR: failed to solve: process "/bin/sh -c pip3 install opencv-python" did not complete successfully: exit code: 1
A possible workaround is to install opencv-python from the alpine repository
FROM python:alpine RUN apk add py3-opencv
but the package is not available to the runtime python distribution (not included in the output of pip list).
$ docker run --rm -it opencv-python pip list Package Version ---------- ------- pip 24.0 setuptools 70.3.0 wheel 0.43.0
File "...Python312\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 77, in build_backend
[...]
File "...Python312\Lib\importlib_init.py", line 90, in import_module
[...]
File "...overlay\Lib\site-packages\setuptools_init.py", line 10, in
import distutils.core
ModuleNotFoundError: No module named 'distutils'Have you tried updating pip, it used to vendor setuptools, which is of course an outdated vendor.
Same issue in our pipelines: microsoft/torchgeo/actions/runs/9356814161/job/25755818665?pr=2103
setuptools/init.py", line 10, in
import distutils.core
ModuleNotFoundError: No module named 'distutils'Have you tried with the environment variable
SETUPTOOLS_USE_DISTUTILSNOT set tostdlib(the other intended valid value islocal, but the implementation details work in a way that any value that's notstdlibshould cause to prefer using the vendored version)Have you tried updating setuptools ? I think it should fallback to its vendored version even with
SETUPTOOLS_USE_DISTUTILS=stdlibIn your own code, make sure you always import setuptools before distutils.
@1riggs Lastly, this PR should also be merged: #1012 , see pypa/setuptools#3935 (comment)
Supporting newer setuptools is great, but is there a reason we don't just remove all references to distutils? The setuptools copy of distutils is also planned to be deprecated, so we're just kicking the can down the road instead of solving the root problem.Oh wait, opencv doesn't use distutils, setuptools does. So yes, I think supporting newer setuptools really is the solution here. Thanks for the fix @bryankaplan and the pointer @Avasam!
Reacted by Avasam and Bryan KaplanI've also encountered this issue today when trying to install opencv on a Windows arm64 machine. So either the build process should be fixed or we could also supply wheels for Windows arm64 architecture.
@CanePlayz I opened #1012 almost four months ago, and it's just sitting there unmerged. Someone reported that it didn't work under MacOS. Can you please test whether it works on your Windows arm64 machine?
Sure, I'll check tomorrow. Since I'm new to this, just to make sure: I'll just have to run
python setup.py bdist_wheelto build the package locally?@CanePlayz Negative; see item 5 under Manual Builds.
Ok, thanks.
I recently fixed a similar Python 3.12+ distutils compatibility issue in OpenDR (PR #522). The root cause is that Python 3.12 removed distutils, and packages that depended on it need updates.
Technical Analysis:
distutils.msvccompiler was removed in Python 3.12. Libraries depending on it need either:- Update to use setuptools instead
- Pin Python version
- Add conditional imports
Solution Pattern (from OpenDR fix):
# Before: from distutils.msvccompiler import MSVCCompiler # After: try: from setuptools._distutils.msvccompiler import MSVCCompiler except ImportError: from distutils.msvccompiler import MSVCCompilerOr simply update the dependency versions to versions that use setuptools instead of distutils.
I've successfully applied this pattern in OpenDR PR #522. Happy to help implement the fix here as well.
I've created a fix for this issue. The problem is that Python 3.12 removed distutils from the stdlib, and the build system was using setuptools<70.0.0 which still relies on internally.
Fix Applied: #1222
The solution updates to require for Python 3.12+. Setuptools 68.0.0+ includes which provides the distutils compatibility layer.
Changes:
- Changed to for Python 3.12+
This follows the same pattern I used in OpenDR PR #522.
Expected behaviour
distutils is deprecated and should not be used in the module.
Actual behaviour
Installing latest version of opencv_python runs into an exception because of the distutils reference.
Call Stack
ERROR: Exception:
Traceback (most recent call last):
File "...Python312\Lib\site-packages\pip_internal\cli\base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\cli\req_command.py", line 245, in wrapper
return func(self, options, args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\commands\install.py", line 377, in run
requirement_set = resolver.resolve(
^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\resolver.py", line 95, in resolve
result = self._result = resolver.resolve(
^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_vendor\resolvelib\resolvers.py", line 546, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_vendor\resolvelib\resolvers.py", line 397, in resolve
self._add_to_criteria(self.state.criteria, r, parent=None)
File "...Python312\Lib\site-packages\pip_vendor\resolvelib\resolvers.py", line 173, in _add_to_criteria
if not criterion.candidates:
^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_vendor\resolvelib\structs.py", line 156, in bool
return bool(self._sequence)
^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\found_candidates.py", line 155, in bool
return any(self)
^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\found_candidates.py", line 143, in
return (c for c in iterator if id(c) not in self._incompatible_ids)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\found_candidates.py", line 97, in _iter_built_with_inserted
candidate = func()
^^^^^^
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\factory.py", line 182, in _make_candidate_from_link
base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\factory.py", line 228, in _make_base_candidate_from_link
self._link_candidate_cache[link] = LinkCandidate(
^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\candidates.py", line 290, in init
super().init(
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\candidates.py", line 156, in init
self.dist = self._prepare()
^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\candidates.py", line 222, in _prepare
dist = self._prepare_distribution()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\candidates.py", line 301, in _prepare_distribution
return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\operations\prepare.py", line 525, in prepare_linked_requirement
return self._prepare_linked_requirement(req, parallel_builds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\operations\prepare.py", line 640, in _prepare_linked_requirement
dist = _get_prepared_distribution(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\operations\prepare.py", line 71, in _get_prepared_distribution
abstract_dist.prepare_distribution_metadata(
File "...Python312\Lib\site-packages\pip_internal\distributions\sdist.py", line 54, in prepare_distribution_metadata
self._install_build_reqs(finder)
File "...Python312\Lib\site-packages\pip_internal\distributions\sdist.py", line 124, in _install_build_reqs
build_reqs = self._get_build_requires_wheel()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\distributions\sdist.py", line 101, in _get_build_requires_wheel
return backend.get_requires_for_build_wheel()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\utils\misc.py", line 745, in get_requires_for_build_wheel
return super().get_requires_for_build_wheel(config_settings=cs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_vendor\pyproject_hooks_impl.py", line 166, in get_requires_for_build_wheel
return self._call_hook('get_requires_for_build_wheel', {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_vendor\pyproject_hooks_impl.py", line 321, in _call_hook
raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
File "...Python312\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 77, in build_backend
obj = import_module(mod_path)
^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\importlib_init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1310, in _find_and_load_unlocked
File "", line 488, in _call_with_frames_removed
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1331, in _find_and_load_unlocked
File "", line 935, in _load_unlocked
File "", line 995, in exec_module
File "", line 488, in call_with_frames_removed
File "...overlay\Lib\site-packages\setuptools_init.py", line 10, in
import distutils.core
ModuleNotFoundError: No module named 'distutils'