ModuleNotFoundError: No module named 'setuptools._distutils'

[knc6]

Hi,

We use readthedocs for the jarvis-tools package. Recently, I am seeing an error due to setuptools:

Raw log here: https://readthedocs.org/api/v2/build/15584283.txt

I am wondering if somebody can help me with this.

---

📌 Edit by @astrojuanlu 📌 This issue is triggered by setuptools 60, released on 2021-12-19. Current status:

• If your project doesn't have a pyproject.toml, most likely you have the "use system packages" version enabled. You can (1) disable it, (2) cap setuptools yourself in your requirements, or (3) wait for us to deploy a fix (January 2022) . See below for more context. Update: This has been fixed on RTD as of Dec 21.
• If your project does have a pyproject.toml, most likely pip is using its fallback mode and installing the problematic setuptools version in an isolated environment. In this case, there is nothing Read the Docs can do to fix your problem. You can do it yourself by adding a proper [build-system] PEP 518 table to your pyproject.toml file. See below for more information.
• You can temporarily roll back to the old setuptools behavior by exporting a SETUPTOOLS_USE_DISTUTILS=stdlib environment variable for your project. However, it is not guaranteed that this will work forever, and we encourage you to look into the solution above.
• If you were redirected here, it's because you opened a new issue for this exact problem. Please try the solution above first.
• If it didn't work, leave a comment below with a link to your failing build.

[bhazelton]

We're seeing the same problem on PR builds for the pyuvdata package.

This is the log:
https://readthedocs.org/projects/pyuvdata/builds/15585777/

I've done some digging, I think this is caused by setuptools version 60.0 first being installed and then downgraded to version 58.2.0. See these related issues:
https://github.com/pypa/setuptools/issues/2939
pypa/setuptools#2353

It seems that this first came up when version 50.0 was released. They fixed it up but then changed it again in 60.0 (with documentation in the changelog). From reading about the issue in 50.0, it seems like this interacts with a bug in pip where the paths get confused when you have one version installed then removed and a different version installed (pypa/setuptools#2353 (comment)).

[knc6]

Interesting, that seems very likely.

I temporarily fixed the issue by getting rid of one of the optional dependencies (phonopy) which was creating the problem.

[astrojuanlu]

Thanks @knc6 for the report and @bhazelton for the help. I have found in your build logs that the line /home/docs/checkouts/readthedocs.org/user_builds/jarvis-tools/envs/develop/bin/python -m pip install --upgrade --no-cache-dir -I mock==1.0.1 pillow==5.4.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.8.1 recommonmark==0.5.0 sphinx<2 sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<2.2 brings setuptools 60.0.0 again, even after doing /home/docs/checkouts/readthedocs.org/user_builds/jarvis-tools/envs/develop/bin/python -m pip install --upgrade --no-cache-dir pip setuptools<58.3.0 right before. I don't think this is a problem with RTD but I want to try and reproduce this, so I'm reopening.

[astrojuanlu]

Oh, of course I understand why. We're passing --upgrade in the second pip install command, so the pip install setuptools<58.3.0 is useless. This will need a fix.

[zaneselvans]

We also seem to be experiencing this problem:

https://readthedocs.org/projects/catalystcoop-pudl/builds/15588383/

[astrojuanlu]

I see that both catalystcoop-pudl from @zaneselvans and jarvis-tools from @knc6 have the "use system packages" option enabled. For this reason, the second pip install command adds a -I, --ignore-installed and that's why the newer setuptools gets installed:

readthedocs.org/readthedocs/doc_builder/python_environments.py

Lines 523 to 528 in c3b4bef

	if self.config.python.use_system_site_packages:
	# Other code expects sphinx-build to be installed inside the
	# virtualenv. Using the -I option makes sure it gets installed
	# even if it is already installed system-wide (and
	# --system-site-packages is used)
	cmd.append('-I')

I anticipated setuptools breakage in #8711 (and probably it's thanks to it that we aren't seeing dozens of user reports) but I didn't realize that the -I flag was being added in some cases.

I'm about to push a fix that hopefully will get deployed soon, but in the meantime, can you test if disabling the "use system packages" option does the trick? And, if you are using it because of some other reason, state why?

Alternatively, you can add a setuptools-requirement.txt pinning setuptools (for example setuptools<58.3.0) and include it in your .readthedocs.yaml config, as explained in https://docs.readthedocs.io/en/stable/config-file/v2.html#requirements-file

[bhazelton]

I checked and we had the "use system packages" option enabled as well, I don't recall why. I unchecked it and re-ran the builds and they passed. So that fixed it for us. Thank you!

[zaneselvans]

We were using the system packages because we depend on Matplotlib, scipy, numpy, etc. and wanted to avoid installing them from scratch. But turning off system packages seems to have fixed the issue.

[uvchik]

We have the same issue but I cannot see that we added explicitly "use system packages". I also tried setuptools<58.3.0 and it worked. During the process setuptool is downgraded to 58.2.0 but this does not solve the problem. Any hints?

Project: oemof/oemof-solph
RTD-log: https://readthedocs.org/projects/oemof-solph/builds/15593968/