Expose enabled extras to build backend

[CGamesPlay]

What's the problem this feature will solve?

Python native extensions cannot take advantage of optional dependencies. This is because the information required to support optional dependencies is not exposed at build time.

For a specific example, I'm investigating adding wasm support to tree-sitter. To do this, I want to add a wasm extra to tree-sitter which enables a dependency on wasmtime. However, I also need to set a compiler flag to enable the functionality in the native extension. It's not possible to always enable the compiler flag, because then users without the extra will get link errors (this isn't quite true, see below).

Describe the solution you'd like

I'd like a mechanism to access the selected extras to be exposed to setup.py, so that it may then alter its build process to enable or disable the extra.

Alternative Solutions

In order to work around this issue, it's necessary for my native extension to create stub methods that can dynamically load the endpoints at runtime (effectively, I have to use dlopen instead of letting the linker do its job).

Additional context

Here's another user from 2022 who encountered the same issue.

I hope this is the right place to request this. Thanks for the consideration.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

This would require a revision to the standardised build backend protocol, and as such would need a PEP. It's not something pip would support without an agreed revision to the standard.

An alternative would be to use a config setting, but config settings are backend-specific, so whether (and how) you could do this would depend on the build backend you're using.

[CGamesPlay]

Thanks for the quick response. Just to evaluate your suggested workaround: config settings appear to only be provided via the command line, which means that they cannot be provided when my project is a dependency of another. Is that correct?

[pfmoore]

Correct. Generally, if you want to allow people to install as a dependency I'd recommend publishing a binary wheel in any case, so that people don't need to have a build environment available. If you publish binaries, the whole question of build options goes away (except for you, when you build the wheel, of course 🙂)

[CGamesPlay]

Publishing binaries makes sense, but I don't think it obviates the problem of optional dependencies. There is a legitimate case to be made for installing with a smaller set of dependencies. It also seems like a subpar workaround, since it effectively says that for architectures I can build for, they get to support the optional dependency, but the rest don't.

I'll stick with using dlopen and a bunch of manually written trampolines for now. Perhaps one day someone will take up writing such a PIP to actually support this. Thanks!

[pradyunsg]

FWIW, making extras fancier/more capable than just a bit of metadata of "a bunch of dependencies" is something that has been discussed in the past: https://discuss.python.org/t/how-to-detect-which-extras-were-installed-at-runtime/15367/4

I think you might be modelling this after Rust's feature flags which is effectively exposed at compile time and allows additional dependencies, whereas Python's extras are literally optional dependencies and nothing more.

[pradyunsg]

I'm gonna go ahead and close this out, given that we already applied a resolution tag here. If a standard is established, we'll adopt it. :)