[BUG] No module named '_sysconfigdata_d_linux_x86_64-linux-gnu'

[henryiii]

setuptools version

setuptools==60.0.0

Python version

Python 3.9

OS

Ubuntu 20.04

Additional environment information

Can reproduce in Docker or with deadsnakes action.

Description

Today, 3.9 deadsnakes debug builds on GitHub Actions for pybind11 started failing with ModuleNotFoundError: No module named '_sysconfigdata_d_linux_x86_64-linux-gnu'. Setting SETUPTOOLS_USE_DISTUTILS=stdlib` fixes it.

Logs: https://github.com/pybind/pybind11/runs/4583094186?check_suite_focus=true

Original report: deadsnakes/issues#188

Expected behavior

It should be possible to use setuptools and sysconfig on Python 3.9 debug builds! :)

How to Reproduce

deadsnakes/action@v2.1.1 on ubuntu-latest, GHA, followed by the final line below. Or it can be reproduced in Docker. Here is a MWE:

$ docker run --rm -it ubuntu:20.04
# apt update && apt install software-properties-common -y
# add-apt-repository ppa:deadsnakes/ppa && apt update && apt install -y python3.9-dev python3.9-dbg python3.9-venv python3.9-distutils
# python3.9-dbg -m venv .venv
# ./.venv/bin/python -m pip install -U pip setuptools wheel
# ./.venv/bin/python -c "import setuptools; from distutils import sysconfig as s; s.get_config_var('LDVERSION')"

Output

Traceback (most recent call last):
  File "/.venv/lib/python3.9/site-packages/setuptools/_distutils/sysconfig.py", line 488, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata_d_linux_x86_64-linux-gnu'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/.venv/lib/python3.9/site-packages/setuptools/_distutils/sysconfig.py", line 601, in get_config_var
    return get_config_vars().get(name)
  File "/.venv/lib/python3.9/site-packages/setuptools/_distutils/sysconfig.py", line 532, in get_config_vars
    func()
  File "/.venv/lib/python3.9/site-packages/setuptools/_distutils/sysconfig.py", line 491, in _init_posix
    _temp = __import__(
ModuleNotFoundError: No module named '_sysconfigdata'

Code of Conduct

• I agree to follow the PSF Code of Conduct

[jaraco]

Looks like that code has been around a while, but it's not present in cpython's copy of distutils. Maybe that's an indication that there are some changes to the stdlib distutils that didn't make it into the standalone repo.

[jaraco]

Hmm. The relevant change is python/cpython#24549 python/cpython#23142. Someone's going to need to figure out how to apply those changes to standalone distutils.

[henryiii]

I'm guessing the problem with a direct port is that it would need to be Python version dependent?

[henryiii]

The latest version here works with Python 3.6+.

No, so was it just never applied?

[henryiii]

Hmm, this is a PR from April, how could it be breaking Python 3.9?

[jaraco]

Thanks Henry for the help. And thanks for creating the PR against distutils! Correct, it was never applied.

Hmm, this is a PR from April, how could it be breaking Python 3.9?

I believe the change was never backported and was only applied to Python 3.10, so may not have received any thorough testing on various Python versions.

[jaraco]

Corrected above, I somehow got confused when bouncing between PRs. The PR I believe to be needed by distutils is python/cpython#23142.

[henryiii]

That PR removes most of distutils.sysconfig and replaces it with sysconfig. Would that be safe for older Pythons? (3.6+). I checked a few, and some of them (like _findvar1_rx expand_makefile_vars) are clearly missing.

[FFY00]

Looks like that code has been around a while, but it's not present in cpython's copy of distutils. Maybe that's an indication that there are some changes to the stdlib distutils that didn't make it into the standalone repo.

I don't think so. I don't have much time to dive into this, but my gut is telling me that the failure is because of the unusual loading that distutils makes of sysconfig. The failure is related to the loading of config variables, the _sysconfig* module is an automatically generated module that includes the Makefile variables, for sysconfig to then load, it is not available on Windows. This is a very private bit of CPython and distutils should not be poking anywhere near it. distutils must be misusing private sysconfig APIs, or there is a bug sysconfig due to the way distutils loads stuff. I am guessing that the answer might actually be something in between.