Overrides in TOML

[henryiii]

I've really been enjoying the design of MyPy's 0.9xx TOML interface. I think the same idea could be applied to cibuildwheel, possibly. Here's my idea of what it would look like:

[tool.cibuildwheel]
# Normal options, etc.
manylinux-x86_64-image = "manylinux2010"

[[tool.cibuildwheel.overrides]]
select = "cp36-*"
manylinux-x86_64-image = "manylinux1"

[[tool.cibuildwheel.overrides]]
select = "cp310-*"
manylinux-x86_64-image = "manylinux2014"

Each overrides item overrides the matching build selected by the expression (analogous to tool.mypy.overrides and module). build could be called something else to differentiate it, or or could be build for similarity. Overrides are applied to every build selector they match sequentially.

This would be really useful in reducing one of the two remaining reasons to use environment variables, and would simply both CI and local builds. The fact that CPython 3.10 can't build with manylinux1 makes this especially useful now (and, really, there's little point in building manylinux1 CPython 3.9 wheels either, even NumPy bumped those up to manylinux2010).

The other remaining reason for environment variables is to build two different wheels from a single identifier (like manylinux1 and manylinux2010), which this would not solve. I think this is special enough that it does not need to be addressed here.

[joerick]

I really like this idea! I had a similar idea in #516 (reply in thread), but it didn't go anywhere because it involved a total overhaul of how the options were parsed. This would fit really neatly inside the existing options structure.

A couple points on the above snippet-

• Should we use select/override rather than build to say what the override refers to? Otherwise it might get confusing with the existing build option...

• Should we allow build/skip in an override? My gut says that it could get messy, but something like this could be useful?

  [[tool.cibuildwheel.overrides]]
  select = "cp36-*"
  skip = "*aarch64"

  ... I think that this shouldn't be allowed, this is better handled by `skip = "cp36-*aarch64".

The other interesting thing to talk about is implementation, because we'd have to delay evaluation of the build options until each build. That's probably fine, but presents an interesting challenge when we print the options at the start of a cibuildwheel session! Maybe there's something we can do like-

Build options:
  platform: 'linux'
  architectures: {<Architecture.x86_64: 'x86_64'>, <Architecture.i686: 'i686'>}
  before_all:
    *: 'yum install libz-devel'
    cp36-*: 'yum install libz-devel && yum install sqlite'
  before_build: 'scripts/prepare'

[henryiii]

Actually, NumPy has bumped 3.10 up to manylinux2014, making this even nicer to have. The default manylinux 2010 build is not able to download NumPy wheels on Python 3.10.

[henryiii]

Also #627 would really benefit from this. I'll try to work on it on the same timescale as that (which looks like it will be really soon, so early next week).

[mayeut]

I'm going to experiment the musllinux build on cmake. This discussion came to mind immediately because I think (not done the work yet), it would be easier with this.

[henryiii]

A few thoughts as I've been working on the implementation. First, before_all and *-image are possibly the most important things to be able to override, and they happen to be special. Overriding the image needs to cause more image launches; you'll need to do one per every unique image name (as is done now, but currently it does not depend on the config). before-all applies per image, so setting a different before-all on a single should either be an error or should also cause more image launches. I think, at least for now, simply verifying it does not vary per-image is fine, and throwing an error if it does.

So this is valid:

[tool.cibuildwheel]
before-all = "yum install somelib"

[[tool.cibuildwheel.overrides]]
select = "cp310-*"
manylinux-x86_64-image = "manylinux_2_24"
before-all = "apt-get install somelib"

and this is an error:

[tool.cibuildwheel]
before-all = "yum install somelib"

[[tool.cibuildwheel.overrides]]
select = "cp310-*"
before-all = "apt-get install somelib"

We could expand this later if needed to add new docker launches if before-all changes (but this already is also how Windows and macOS would behave - there are no containers there, so they support only one before-all).

[joerick]

Ah, yes, I hadn't foreseen this complication. Good point. Just so I'm sure I understand the examples above, are you saying that it will be valid to change the image for a specific Python version? Quite a big refactor! I'm guessing you have a use-case in mind for this?

[henryiii]

Yes, the one listed above. NumPy is only shipping manylinux 2014 wheels only for Python 3.10 (since all versions of Pip that support Python 3.10 understand manylinux2014, it's a sensible thing to do), so most (all?) data-science packages will likely want to do manylinux2010 for Python 3.6-3.9, and manylinux2014 for 3.10+. Or manylinux1 for 3.6-3.8, manylinux2010 for 3.9, and manylinux2014 for 3.10+. Even without requiring NumPy for either build or test, this is still a good idea - there's little point in releasing manylinux2010 wheels for Python 3.10, unless you really do need CentOS 6 compatible wheels. On Python 3.10....

If we can't trigger splitting based on this right away, we can also make it an error for now - but I think it will be pretty easy to split by image. Let's see.

[JesseFarebro]

Is environment variable parsing supported using the TOML overrides? There's something weird going on when I try to export a host variable, e.g.,

[[tool.cibuildwheel.overrides]]
select = "*-manylinux_x86_64"

  [tool.cibuildwheel.overrides.environment]
    MY_VAR = "$MY_VAR"

But it doesn't seem to be exported properly.

Also, has it ever been discussed how to have a method to extend the set of environment variables? I wish the environment variables were a list of tables so I could easily append environment variables. With the current API I have copies of five or six environment variables for each override.

[henryiii]

That should be a TOML error (as in, someone needs to throw an error). Overrides is an array. You have to use the nested syntax.

[[tool.cibuildwheel.overrides]]
select = "*-manylinux_x86_64"
environment = { MY_VAR = "$MY_VAR" }

Due to (arbitrary) TOML restrictions, you cannot place a new line or an ending comma here. You can use a string if you prefer.

[henryiii]

The problem with extending an environment is that there's no way to undo it. You could set a variable to empty string, but you can't remove a variable. That said, I think I would be OK with extending rather than replacing by default, but you would lose the ability to do that. It should even be backward-compatible unless you were relying on a variable not being set.

[JesseFarebro]

If you put this into a TOML parser, e.g., http://toml-online-parser.ovonick.com

[[tool.cibuildwheel.overrides]]
select = "*-manylinux_x86_64"

  [tool.cibuildwheel.overrides.environment]
    MY_VAR = "$MY_VAR"

[[tool.cibuildwheel.overrides]]
select = "*-macos_x86_64"

  [tool.cibuildwheel.overrides.environment]
    MY_VAR_2 = "$MY_VAR"

it gets parsed the same as what you suggest. It does look unusual, I just wanted a way to have environment variables over multiple lines. Inline tables can't be written over multiple lines. If we did have the notion of extending the environment I could just specify the environment table and override a single environment variable per override selection. Right now I'm stuck copying this to all my overrides.

RE: extending environment. In a lot of cases setting the env variable to an empty string would be sufficient, shell scripts at least usually use test -n. Another option might be to have a special key like "+environment" or something which would override the environment completely and not inherit anything.

[JesseFarebro]

@henryiii there's definitely something going on with parsing environment variables from overrides. I tried forwarding some of the Github environment variables like GITHUB_REF and they just all get parsed as an empty string. The cibuildwheel config shows:

cp37-manylinux_x86_64: ParsedEnvironment(['GITHUB_REF="${GITHUB_REF}"'])

I've also tried $GITHUB_REF but everything gets replaced with the empty string.

Edit: I don't see how it's possible that it's not parsing, perhaps I must be missing something.

Edit2: I see, this doesn't work as I expected. It won't evaluate this in the host environment so I can't pass GITHUB_REF from the host. Without the ability to extend environment variables I'm not sure how I'm supposed to access environment variables from the host by just using pyproject.toml. The way I was doing this before was using the GHA sugar ${{ env.BLAH }} to pass it right into CIBW_ENVIRONMENT. This isn't possible from pyproject.toml and if I set CIBW_ENVIRONMENT I lose all my environment variables from pyproject.toml.

Has there been any work on exporting host environment variables?

[bldrvnlw]

@JesseFarebro I have the same problem and so migrated all the Github environment variables into the pyproject.toml.

However this showed the issue that environment settings are not merged so environment set at say [tools.cibuildwheel] level is not merged with environment at [tools.cibuildwheel.linux] but only the settings at the more specific level are visible. As a result the pyproject.toml now has error prone duplication.

As well as overrides a merge classifier would provide a solution

e.g.

[tool.cibuildwheel.environment]
ALICE="sender"

[tool.cibuildwheel.linux.merge.environment]
BOB="listener"

[tool.cibuildwheel.macos.merge.environment]
EVE="eavesdropper"

Resulting in the following environment variables defined on the two platforms
linux : ALICE BOB
macos: ALICE EVE

To return variables to the Github environment - it is would be nice to have a cibuildwheel function that could be used to easily pluck defined values (possibly there is one as I'm unfamiliar with the internals). Then a GitHub step could read and set pyroject.toml vaules as outputs for other steps or jobs in Github

If I'm missing some trick I'd be glad to hear it