[BUG] `AttributeError: 'PosixPath' object has no attribute 'read_text'` with setuptools 60.9.0+ and `pathlib`

[edmorley]

setuptools version

setuptools==62.1.0 (however the earliest affected version is setuptools==60.9.0)

Python version

Python 3.9 (though 3.7 and 3.8 are also affected, plus 3.10 is affected but gives a different error message)

OS

Docker image python:3.9.12 which is Debian 11 (it also occurs on Ubuntu on Heroku)

Additional environment information

mrio-common-metadata==0.2

(which indirectly pulls in pathlib==1.0.1)

Description

Between setuptools 60.8.2 and 60.9.0, a previously working project now encounters an error during the python setup.py egg_info phase of pip install:

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-ymj9drfq/pathlib_99a5e0894a554de3b2297b3e6602e3f7/setup.py", line 6, in <module>
          setup(
        File "/usr/local/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 109, in setup
          _setup_distribution = dist = klass(attrs)
        File "/usr/local/lib/python3.9/site-packages/setuptools/dist.py", line 460, in __init__
          for ep in metadata.entry_points(group='distutils.setup_keywords'):
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 999, in entry_points
          return SelectableGroups.load(eps).select(**params)
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 449, in load
          ordered = sorted(eps, key=by_group)
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 997, in <genexpr>
          dist.entry_points for dist in unique(distributions())
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 609, in entry_points
          return EntryPoints._from_text_for(self.read_text('entry_points.txt'), self)
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 917, in read_text
          return self._path.joinpath(filename).read_text(encoding='utf-8')
      AttributeError: 'PosixPath' object has no attribute 'read_text'

The changes between those two setuptools versions are:
v60.8.2...v60.9.0

Under Python 3.7/3.8 the failure is the same as seen on Python 3.9 above.

However under Python 3.10, the error changes to:

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

Downgrading to setuptools==60.8.2 resolves the issue.

It seems that the pathlib package in the environment is being used by setuptools instead of the module in the stdlib. This should not happen, since the stdlib has higher precedence on sys.path than site-packages.

See also previously filed #3132. I've filed this separately, since:

• that issue is closed (I believe incorrectly, due to a misunderstanding about sys.path precedence)
• the steps to reproduce here are slightly different (a real world case encountered in a customer app)
• this issue has more details than the other one

...however feel free to dupe over there and reopen that issue if preferred :-)

Expected behavior

Either:

• Installing mrio_common_metadata==0.2 continues to work as it did for setuptools <= 60.8.2.
• Or, as a major version release (not a minor version bump), setuptools causes this scenario to be an error with an easy to debug error message.

How to Reproduce

For the Python 3.9 repro:

1. docker run --rm -it python:3.9.12 bash -c 'pip install pip==22.0.4 setuptools==62.1.0 && pip install mrio_common_metadata==0.2'

For the Python 3.10 alternative error message variant:

1. docker run --rm -it python:3.10.4 bash -c 'pip install pip==22.0.4 setuptools==62.1.0 && pip install mrio_common_metadata==0.2'

Output

Python 3.9:

$ docker run --rm -it python:3.9.12 bash -c 'pip install pip==22.0.4 setuptools==62.1.0 && pip install mrio_common_metadata==0.2'
...
Successfully installed setuptools-62.1.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Collecting mrio_common_metadata==0.2
  Downloading mrio_common_metadata-0.2.tar.gz (12 kB)
  Preparing metadata (setup.py) ... done
Collecting pyxlsb
  Downloading pyxlsb-1.0.9-py2.py3-none-any.whl (23 kB)
Collecting pathlib
  Downloading pathlib-1.0.1.tar.gz (49 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.3/49.3 KB 24.8 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-ymj9drfq/pathlib_99a5e0894a554de3b2297b3e6602e3f7/setup.py", line 6, in <module>
          setup(
        File "/usr/local/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 109, in setup
          _setup_distribution = dist = klass(attrs)
        File "/usr/local/lib/python3.9/site-packages/setuptools/dist.py", line 460, in __init__
          for ep in metadata.entry_points(group='distutils.setup_keywords'):
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 999, in entry_points
          return SelectableGroups.load(eps).select(**params)
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 449, in load
          ordered = sorted(eps, key=by_group)
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 997, in <genexpr>
          dist.entry_points for dist in unique(distributions())
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 609, in entry_points
          return EntryPoints._from_text_for(self.read_text('entry_points.txt'), self)
        File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 917, in read_text
          return self._path.joinpath(filename).read_text(encoding='utf-8')
      AttributeError: 'PosixPath' object has no attribute 'read_text'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

Python 3.10:

$ docker run --rm -it python:3.10.4 bash -c 'pip install pip==22.0.4 setuptools==62.1.0 && pip install mrio_common_metadata==0.2'
...
Successfully installed setuptools-62.1.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Collecting mrio_common_metadata==0.2
  Downloading mrio_common_metadata-0.2.tar.gz (12 kB)
  Preparing metadata (setup.py) ... done
Collecting pyxlsb
  Downloading pyxlsb-1.0.9-py2.py3-none-any.whl (23 kB)
Collecting pathlib
  Downloading pathlib-1.0.1.tar.gz (49 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.3/49.3 KB 24.8 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

[abravalheri]

Hi @edmorley, I think the situation is a bit more complicated than that.

The pathlib package on PyPI does not contain a pyproject.toml file, which will make pip use the setuptools.build_meta:__legacy__ backend as specified in PEP 517.

To install pathlib, setuptools.build_meta:__legacy__ will run the equivalent of python setup.py bdist_wheel from inside the directory containing the pathlib code. This will effectively put the outdated/deprecated/discouraged implementation of pathlib as the first entry in the Python path, isn't it?

Therefore I believe that the explanation for the 3131 being closed is still valid, do you agree?

[edmorley]

@abravalheri Hi! Thank you for taking a look and your quick reply :-)

I'm not familiar with how the setuptools.build_meta:__legacy__ backend is meant to work, however my initial thoughts are:

1. The fact that stdlib normally shadows any installed site-packages are a fundamental part of the Python ecosystem with respects to the way backport packages and similar are used. As such, any design that doesn't take this into account, seems prone to causing breakages and thus perhaps needs revisiting?
2. This package installed fine with earlier versions of setuptools, and the failure only started occurring in a minor version setuptools release with no mention in the changelog and an unclear error message - so seems unintended to me?

As such, my questions are:

1. Was this behaviour change intended / desired?
2. If it is intended, should it instead be reverted temporarily, and then rolled out in a new major version after a deprecation period/warnings etc?
3. Or, if it is intended but it's too late to revert/roll out more slowly, can the failure mode at least be made easier to debug?

Many thanks :-)

[edmorley]

I'm presuming one workaround (that doesn't involve pinning setuptools, which is not viable long term), would be to get the author of pathlib to publish a wheel for the latest version? That way setuptools never needs to be invoked during its installation?

Albeit, such a workaround would not help any other packages that are either backports or older versions of things now in the stdlib, that may cause similar issues.

[abravalheri]

Hi @edmorley,

Regarding the questions, I think the short answer is that setuptools needs to rely that any public APIs exposed by the standard library is honoured.

I think we can safely say that this reliance is intentional and desired.

The API and behaviour of setuptools.build_meta:__legacy__ is also intentional and desired, otherwise other old packages in the ecosystem may break.

It would be extremely complicated to revert the changes at this point (they brought a bunch of fixes for other issues people have been facing and improvements that people started to rely on in the last months).

Regarding the concerns with other backports that might happen to use setuptools, we can safely assume the following:

1. During normal execution, the sys.path ordering will make sure stdlib has higher precedence, therefore setuptools will only be affected when building that particular backport from source or sdist.
2. This means that, if a backport maintenance is abandoned, the authors should either:
   • publish wheels, this way the build phase is skipped
   • pin the version of setuptools being used for their build via pyproject.toml [build-system]
     If none of these measures are taken, it is a sign that the authors are OK with the abandoned package not being able to be built.

can the failure mode at least be made easier to debug?

I honest don't know what can be done to make the situation more easy to debug. Setuptools is composed by several sub-packages/modules and currently bundles a few dependencies. All of them can use pathlib or any other package/module being provided by the stdlib.

It would be an immense cognitive overhead to double check all potentially non-backported stdlib APIs before calling them and we would also need to coordinate the same to be done by all bundled dependencies.

If you have any suggestion on how to do that, please feel free to submit a PR and we will be more than happy to have a look.

[abravalheri]

Just a other comment:

The fact that stdlib normally shadows any installed site-packages are a fundamental part of the Python ecosystem with respects to the way backport packages and similar are used. As such, any design that doesn't take this into account, seems prone to causing breakages and thus perhaps needs revisiting?

I don't think the stdlib shadows the first entry of sys.path that corresponds to CWD (automatically added), does it? If we look at the problem by this optics we can say that setuptools behaviour is not anomalous, but instead exactly what is expected in the Python ecosystem.

This behaviour is intentional and changing it might make other packages to break. The good news is that the majority of the (still supported) packages will use setuptools.build_meta which does not add an entry to sys.path.

[edmorley]

I don't think the stdlib shadows the first entry of sys.path that corresponds to CWD (automatically added), does it? If we look at the problem by this optics we can say that setuptools behaviour is not anomalous, but instead exactly what is expected in the Python ecosystem.

Correct, CWD has higher precedence even than the stdlib. I'm not proposing that that isn't the case or should be changed :-)

Instead, what I'm pointing out is that an existing behaviour of setuptools has:

• changed in a breaking way (it's now invoking installs in a way that means the CWD is that package or similar)
• in a minor version (we were pinned to a specific version; I mention this more since it suggests the change wasn't expected)
• with no changelog entry or warnings.

I totally get that in order to improve setuptools it's necessary to make breaking changes.
And it sounds like you are confirming this new behaviour is not a bug?

However even after reading the replies above, I still can't quite work out whether this side effect (of the way setuptools invoked things meaning the CWD now that on the package, so it is on sys.path and thus causing the breakage mentioned in the OP):

1. Was known and a conscious design decision of one of the changes in v60.8.2...v60.9.0, but was (for unknown reasons) just not documented in the changelog and determined fine to include in a minor version.
2. Was not anticipated (and so why not documented in the changelog etc), but rather a side-effect of other work, and is being left as-is because it also happens to be desired behaviour.

I'd like to hope (2) :-)

[abravalheri]

Hi @edmorley,

The way the build/install works with setuptools was not changed. CWD has always being added to sys.path on setuptools.build_meta:__legacy__ (some packages depend on this behaviour).

What happened is that one of the bundled dependencies is now using pathlib. I don't think it is fair to classify this as a breaking change since pathlib is part of the standard library and as such is supposed to relied upon.

This change in the bundled dependencies is intentional to allow setuptools to eventually remove pkg_resources which will make the project much faster and less error prone.

What are your suggestions going forward? I think we can add a warning to the CHANGELOG specially for packages depending on pathlib.

[edmorley]

Ah this was the piece that was missing from above:

What happened is that one of the bundled dependencies is now using pathlib

With that, I follow now - thank you :-)

(Before it seemed that there had been an intentional setuptools change to CWD handling or something, rather than a latent issue being exposed by a dependency change etc)

[abravalheri]

Going forward other parts of setuptools might also start importing pathlib. I don't think we want to prevent that from happening.