Presence of a pyproject.toml file causing unexpected behavior

[omry]

pip version

21.0.1

Python version

3.8.0

OS

Ubuntu 18.04

Additional information

No response

Description

Somehow, the presence of a pyproject.toml file is causing pip install from a directory to fail importing of installed modules.

Expected behavior

The presence of pyproject.toml does not impact the behavior of pip install.

How to Reproduce

This is a bit complicated to explain minimal repro here should be enough.

Structure:

$ tree
.
├── build_helper
│   ├── build_helper
│   │   └── __init__.py
│   └── setup.py
├── example
│   └── __init__.py
├── README.md
└── setup.py

Content of files in repro repository.

Output

Installing build_helper:

$ pip install build_helper/
Processing ./build_helper
Building wheels for collected packages: build-helper
  Building wheel for build-helper (setup.py) ... done
  Created wheel for build-helper: filename=build_helper-0.0.0-py3-none-any.whl size=1261 sha256=aa0ee67ef3385cba13e32659fb3635047631d0466ef0861c71badd5d3d67f7fd
  Stored in directory: /tmp/pip-ephem-wheel-cache-_dpz2jwm/wheels/36/26/ae/14e3035420d431e16878ed266d823b24627dc0fe56af0b53b3
Successfully built build-helper
Installing collected packages: build-helper
  Attempting uninstall: build-helper
    Found existing installation: build-helper 0.0.0
    Uninstalling build-helper-0.0.0:
      Successfully uninstalled build-helper-0.0.0
Successfully installed build-helper-0.0.0

Successfully installing primary project:

$ pip install .
Processing /home/omry/dev/test_pip
Building wheels for collected packages: example
  Building wheel for example (setup.py) ... done
  Created wheel for example: filename=example-1.0.3-py3-none-any.whl size=991 sha256=21a95c7e50946865cb6f4565842e455ab9a972ceae1800e534d391e2032286c0
  Stored in directory: /tmp/pip-ephem-wheel-cache-k02av0q_/wheels/57/c2/45/2447e9fc2acaa90e70563527f393553ccc1297fb8d529f0c92
Successfully built example
Installing collected packages: example
  Attempting uninstall: example
    Found existing installation: example 1.0.3
    Uninstalling example-1.0.3:
      Successfully uninstalled example-1.0.3
Successfully installed example-1.0.3

Creating an empty pyproject.yaml, after which installation fails:

$ touch pyproject.toml
$ pip install .
Processing /home/omry/dev/test_pip
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /home/omry/miniconda3/envs/test-pip/bin/python /home/omry/miniconda3/envs/test-pip/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpybcsa03y
       cwd: /tmp/pip-req-build-_31oxf1l
  Complete output (18 lines):
  Traceback (most recent call last):
    File "/home/omry/miniconda3/envs/test-pip/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/home/omry/miniconda3/envs/test-pip/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/omry/miniconda3/envs/test-pip/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 114, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-m265pb85/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 154, in get_requires_for_build_wheel
      return self._get_build_requires(
    File "/tmp/pip-build-env-m265pb85/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 135, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-m265pb85/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 258, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/tmp/pip-build-env-m265pb85/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 150, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 3, in <module>
      from build_helper import get_version
  ImportError: cannot import name 'get_version' from 'build_helper' (unknown location)
  ----------------------------------------
WARNING: Discarding file:///home/omry/dev/test_pip. Command errored out with exit status 1: /home/omry/miniconda3/envs/test-pip/bin/python /home/omry/miniconda3/envs/test-pip/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpybcsa03y Check the logs for full command output.
ERROR: Command errored out with exit status 1: /home/omry/miniconda3/envs/test-pip/bin/python /home/omry/miniconda3/envs/test-pip/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpybcsa03y Check the logs for full command output.

Code of Conduct

• I agree to follow the PSF Code of Conduct

[pfmoore]

If you add a pyproject.toml, you need to include build_helper as a build dependency in that file. Projects with pyproject.toml present are processed using PEP 517 rules, which means that they are built in an isolated environment with only the dependencies mentioned as build requirements in pyproject.toml available.

[omry]

I see.
This caught me off guard because I had pyproject.toml (for towncrier) - and things didn't behave as I expected.
It took some binary on which files I should delete to make things work. pyproject.toml was the last file I tried.

Is this behavior documented in the pip docs? I actually tried to look for build time dependencies and failed to find anything helpful.

[omry]

I tried this:

$ cat pyproject.toml 
[build-system]
requires = ["build_helper"]

But I couldn't figure out how to specify a local dependency.

Based on this SO question, it was not supported 8 months ago.

As a fallback, is there a way to get pip to ignore the pyproject.toml file?

[pfmoore]

It's mostly documented in PEPs 517 and 518, as those are the relevant standards and pip implements standard defined behaviour. There's no standard-compliant way to specify a build dependency as a local file relative to the source directory, no.

I'm assuming that putting your towncrier configuration in a file other than pyproject.toml isn't possible? That would be the simplest solution until you're ready to transition to pyproject.toml for your build configuration.

But assuming not, then there are some short-term alternatives. I describe these as "short term", but there's no immediate plan for pip to remove them - it's just that ultimately, pip (and the packaging ecosystem in general) will only support PEP 517/518, and at that point you'll need to have found a standard way of implementing your workflow. But for now, you can probably use --no-build-isolation to get the behaviour you want. Alternatively, --no-use-pep517 will disable all PEP 517 processing - but that definitely will be getting removed at some point, as it's a transition mechanism.

[omry]

Thanks for your answers @pfmoore.

Black is also using pyproject.toml for configuration. It does support specifying a path in the command line but it's not very friendly to people running black manually.
I don't think towncrier supports an alternative config name though.

Using one of the two plugins is reasonable workaround for now.
Is there a way to make pip use them by default in a project? (maybe something in pyproject.toml? :)
Otherwise I am pretty sure I will receive many complaints about people telling me my project doesn't install from source.

It looks like pip is using the presence of pyprojects.toml as the gate for the pep517 logic.
Is there a possibility of using the presence of any build specific config section in pyprojects.toml to trigger that logic instead?
pyproject.toml is gaining adoptions by many other tools. This will probably become a bigger problem in the medium term (until the ecosystem have transitioned to the new way of configuring the build).

My use case is as follows:
I have a repository that contains a core section and a plugins section (the core and each of the plugins are packages and distributed).
I want to have some shared logic in the build of the plugins and the core.
That build_helper is my attempt at implementing such reuse. I would rather not be forced to distribute build_helper as it can make things pretty complicated.

1. Are there plans to eventually support local build dependencies?
2. Are there better practices / examples I can follow to achieve something like this?

[pradyunsg]

It looks like pip is using the presence of pyprojects.toml as the gate for the pep517 logic.
Is there a possibility of using the presence of any build specific config section in pyprojects.toml to trigger that logic instead?

No, there isn't. This was an intentional design choice to make downstream packages to start adopting the newer packaging practices, such as build isolation and PEP 517, when they try to use the newer standards for the same.

There have been requests for adding such an escape hatch for a while now, and the answer to those is usually to figure out how to make that package work within the model for package builds that we want in the future. As it stands, you can't have the nice things provided by PEP 518 and PEP 517 (pyproject.toml) without spending some effort to make sure your package is compatible with how the newer build process works (build isolation, PEP 517 backends etc). We haven't so far seen a compelling reason to allow using the niceties of the new standards without moving away from practices that we're pushing people away from.

That build_helper is my attempt at implementing such reuse. I would rather not be forced to distribute build_helper as it can make things pretty complicated.

You can use https://www.python.org/dev/peps/pep-0517/#in-tree-build-backends, if what you want to do is have code used only for performing builds. Note that you'd need to wrap an existing PEP 517 build backend for this and I don't think the setuptools backend would be easy to wrap.

[pfmoore]

Is there a way to make pip use them by default in a project?

No there isn't, sorry.

It looks like pip is using the presence of pyprojects.toml as the gate for the pep517 logic.

The wording from PEP 517 is as follows:

If the pyproject.toml file is absent, or the build-backend key is missing, the source tree is not using this specification, and tools should revert to the legacy behaviour of running setup.py (either directly, or by implicitly invoking the setuptools.build_meta:legacy backend).

What pip is doing if the build-backend key is missing, is to use the setuptools.build_meta:__legacy__ backend, which is not 100% compatible with running setup.py directly (as you've found).

Using --no-build-isolation still uses the backend, but does not use build isolation, which is what won't work with your source-relative build helper.
Using --no-use-pep517 reverts to the direct setup.py approach.

Are there plans to eventually support local build dependencies?

No. Doing so would need changes to the standards, and I'm not aware anyone has proposed any such changes. I don't think it's a use case that is sufficiently widely used to have come to the attention of anyone willing to propose a standard (which, just to be clear, means "anyone" - there's no barrier on proposing standards, other than the requirement to be willing to put in the work).

Are there better practices / examples I can follow to achieve something like this?

Not that I know of, sorry. Maybe someone else can offer suggestions. I think most people would distribute the build helper - but that's just a guess and as you say may not be appropriate for you.

[pradyunsg]

What pip is doing if the build-backend key is missing, is to use the setuptools.build_meta:__legacy__ backend, which is not 100% compatible with running setup.py directly (as you've found).

Wait, using the __legacy__ backend is 100% the same as invoking setup.py (minus going through PEP 517, rather than calling setup.py directly) -- the thing that's messing up this user's build process is the build-isolation related issues.

[piotr-dobrogost]

Is there a possibility of using the presence of any build specific config section in pyprojects.toml to trigger that logic instead?

See discussion in #8437 and especially this comment #8437 (comment)