Passing user-specified arguments to python -m build or python -m pip wheel.

[andyfaff]

Description

I was to build scipy wheels using cibuildwheel. scipy uses mesonpy as a build backend, which then uses meson and ninja. I've got to the stage where I want to cross compile for macosx_arm64. To do so means configuring meson for cross compilation.

I think this can be done via:

# specifies the build directory and configures for cross-compile.
meson setup --cross-file $PROJECT_DIR/tools/wheels/cross_arm64.txt build 
# the builddir option is passed to mesonpy
python -m build -Cbuilddir=build

Specification of the build directory is important because it contains all of the info on the build setup. If builddir is not specified then mesonpy builds in a different directory, all of the specification for cross compilation is lost, and only a native build occurs.

I propose that it should be possible for cibuildwheel to pass options to the front-ends; in this case -Cbuilddir=build to build (or --config-settings builddir=build with pip)?
This has come up a few times in very similar issues, such as #268, #766

Build log

No response

CI config

No response

[andyfaff]

The suggestion you have seems to be pip or build agnostic, and you could provide any possible flags to the front end. Provide other flags with spaces.

CIBW_BUILD_FRONTEND_FLAGS="-Cbuilddir=dir -O -T -H -E -R -F -L -A -G -S"

What was your other suggestion?

[joerick]

Yeah, this seems reasonable to me. The only question I have is whether the option should be CIBW_BUILD_FRONTEND_FLAGS i.e. a string of args which would be passed to the build frontend on the command line like CIBW_BUILD_FRONTEND_FLAGS=-Cbuilddir=build or should we lean into the PEP517 config settings dictionary, with an option like CIBW_BUILD_CONFIG_SETTINGS?

CIBW_BUILD_CONFIG_SETTINGS would be great in TOML, where we can do data structures natively, so it would be like:

build-config-settings = { builddir = "build" }

# or even

[tool.cibuildwheel.build-config-settings]
builddir = "build"

...but, it could be clumsy in an environment variable, since config-settings is theoretically a mix of dicts and arrays. However, in practise, pip only allows to pass dict[str, str], and build allows dict[str, str|list[str]]. It's confusing to me why the structure of config-settings is so limited in both these tools, in contrast to the PEP.

This does give us an easy answer for an environment variable version of CIBW_BUILD_CONFIG_SETTINGS, though, we could mimic build's style a-la CIBW_BUILD_CONFIG_SETTINGS: "name=val some_array=1 some_array=2", which would result in {"name": "val", "some_array": ["1", "2"]}.

As for the idea of CIBW_BUILD_FRONTEND_FLAGS, I'm less well inclined to it, I think. While it does add a lot of flexibility, it makes pip/build's command-line API part of our API, and it would make it possible for users to break the build in unforeseeable ways. If we wanted to add an unlimited freedom option, I'd rather go all-out with an option like CIBW_BUILD_FRONTEND=custom: your-build-frontend --any-setting {package_dir} {built_wheel_dir}

But I'm curious what others think, too.

[andyfaff]

Any of those options would work for us. I like the idea of CIBW_BUILD_CONFIG_SETTINGS, either in toml or environment variable.

[henryiii]

With CIBW_BUILD_FRONTEND=custom: stuff you'd have to also teach cibuildwheel how to download and pin the frontend, which we don't do (though I guess you could do it via pipx). Yuck.

I like CIBW_BUILD_CONFIG_SETTINGS a lot, or maybe CIBW_CONFIG_SETTINGS since tool.cibuildwheel.build-config-settings is already long and "build"ing is all we do. {"a": "b"} in TOML can already convert to a=b as an envvar, so we'd just need to teach it about expanding as a list. I'm guessing this would fail to work if a list is given & the backend is not build, but that's fine for now - just needs a comment.

[ethanhs]

I'm currently working on moving mypy to build mypyc compiled wheels with PEP 517 and I think we are currently blocked on being able to pass --use-mypyc to CIBW, so very interested in seeing this added! I think any of the proposals seem reasonable, but I expect most people will be most familiar with passing command line flags for the config-settings, since that is what they do locally.

[andyfaff]

I forgot to say that I'd like to send the CIBW_CONFIG_SETTINGS to pip as well as build.

[henryiii]

This is merged now & out in 2.10, and I've been trying to get projects to try it out, FYI. ;) Remember it can be mixed with [[tool.cibuildwheel.overrides]]!

[andyfaff]

Thanks for #1244. Given that cirrusci can do native macOS for arm64, we may use that instead of all the bother with cross compiling. In the event we don't we'll use the changes made in #1244