Create 3.14t wheels by default

[ngoldbaum]

Over in gcanat/video_reader-rs#70 I noticed that video_reader-rs wasn't uploading cp314t wheels.

It looks like they're using a vanilla version of the action with no customization about building an explicit python version:

https://github.com/gcanat/video_reader-rs/blob/da2c7c905bdf324a9dca332060e410188f9eca39/.github/workflows/build.yml#L52-L63

That makes me suspect this is coming down to the defaults in maturin-action. Is that the case?

[2bndy5]

Hey @ngoldbaum, fancy seeing you here... I came here looking to ask a similar question. Please bear with me as I suspect you already know most of this. I'm just posting to summarize the blockers I see toward enabling free-threaded builds automatically.

It seems that the project in question used your linked example to enable 3.14t builds:

       - name: Build wheels
         uses: PyO3/maturin-action@v1
         with:
           target: ${{ matrix.platform.target }}
-          args: --release --out dist --find-interpreter
+          args: --release --out dist --interpreter '3.9 3.10 3.11 3.12 3.13 3.13t 3.14 3.14t pypy3.9 pypy3.10 pypy3.11'

But this ultimately depends on the env having the specified versions (namely 3.13t and 3.14t in this case) present. So, those trying to opt-into free-threaded builds should really be mindful about the python versions (and corresponding cpy dev headers) available for the targeted platform.

• The pypa/manylinux docker images seem to have 3.13t and 3.14t present; not sure about musllinux images though.
• But for Windows and MacOS, the build env might need some finesse involving actions/setup-python. For Windows, Environment colision between regular and free-threaded builds when using the Windows installer python/cpython#127294 requires free-threaded builds to happen in a dedicated job (separate from non-free-threaded builds).

This inquiry is further complicated by abi3 wheels. As PyO3/maturin#2772 points out, maturin currently (at this time) lacks the intuition to make separate free-threaded builds when a abi3 feature is enabled. This intuition already exists for PyPy builds when a abi3 feature is enabled, so its not an unrealistic expectation.

I also suspect that maturin --find-interpreter may also need to account for free-threaded versions of Python.

Conclusion

As for making free-threaded builds automatically part of the default maturin-action behavior, I don't think this is currently achievable. Just as a reminder, free-threaded builds are still an explicit opt-in capability in pyo3 (see pyo3 docs). I'm guessing that maturin would have to

1. somehow detect when all modules have the Py_MOD_GIL slot set to Py_MOD_GIL_NOT_USED
2. trust (or "assume" depending on your perception) that all #[pyclass]es are thread safe.

This would probably be more easily done if/when free-threaded support becomes enabled by default (something the current pyo3 docs imply is planned for the future).

[ngoldbaum]

As for making free-threaded builds automatically part of the default maturin-action behavior, I don't think this is currently achievable. Just as a reminder, free-threaded builds are still an explicit opt-in capability in pyo3 (see pyo3 docs). I'm guessing that maturin would have to somehow detect when all modules have the Py_MOD_GIL slot set to Py_MOD_GIL_NOT_USED
trust (or "assume" depending on your perception) that all #[pyclass]es are thread safe.
This would probably be more easily done if/when free-threaded support becomes enabled by default (something the current pyo3 docs imply is planned for the future).

I don't understand this logic. What's wrong with building wheels that reenable the GIL on import (i.e. the pymodules don't declare support)?

Cibuildwheel currently builds 3.14t wheels unless users explicitly opt out. This means that some projects uploaded wheels that do not declare free-threaded support. IMO that's fine since it still fixes build issues for users who want to experiment. I think errors caused by a missing build environment are strictly less user friendly than a runtime warning telling users a module doesn't support the free-threaded build. Even more so because there's an escape hatch for the GIL being reenabled at runtime: set PYTHON_GIL=0.

[2bndy5]

What's wrong with building wheels that reenable the GIL on import (i.e. the pymodules don't declare support)?

This could be described as "unexpected behavior" by consumers that try experimenting with free-threaded Python.

Maturin still needs some proper way of detecting when the GIL is disabled (disregarding thread-safety for classes), even if only to output a warning in the build log.

Cibuildwheel currently builds 3.14t wheels unless users explicitly opt out.

I was aiming for correctness. I wasn't comparing the build experience/process to cibuildwheel.

Much of what was stated after this sentence seems opinionated, so I'll just leave that as a difference of opinions.

escape hatch for the GIL being reenabled at runtime: set PYTHON_GIL=0.

Would that cause an error for modules that don't properly support free-threaded Python? I've never actually tried this.

I imagine errors related to classes that are not thread safe would be much less helpful.

[2bndy5]

pyo3 v0.28.0 has officially made use of the GIL opt-in. Meaning the default has changed to not use the GIL: See pyo3 migration guide.

I'm still not sure how this helps maturin identify free-threaded builds, but it seems to set a precedent.

[ngoldbaum]

I'm still not sure how this helps maturin identify free-threaded builds, but it seems to set a precedent.

FWIW, maturin has supported the free-threaded build for quite a while.

I'm still confused by your hesitation around the GIL being re-enabled at runtime and that leading to users having bad experiences. IMO forcing them to build extensions is an even worse experience.

You asked this a while ago and I guess I missed it:

Would that cause an error for modules that don't properly support free-threaded Python? I've never actually tried this.

Only if they're thread-unsafe or rely on the GIL for thread safety. For most PyO3 extensions this is not the case. Setting PYTHON_GIL=0 is equivalent to opting into experimental "I-know-what-I'm-doing" mode and are OK with possible breakage. I wouldn't suggest anyone doing it in production but it's very valuable for experimentation and adding real support for the free-threaded build.

Anyway, with PyO3 0.28 there will be even fewer situations in practice where rust extensions enable the GIL at runtime.

Are you perhaps OK with adding 3.14t wheel builds to the default set of builds done by maturin-action?

[2bndy5]

Are you perhaps OK with adding 3.14t wheel builds to the default set of builds done by maturin-action?

Sure. But I'm not a deciding factor. I only landed on this thread to see why maturin-action doesn't produce free-threaded builds (where applicable).

[ngoldbaum]

Good point. Maybe @messense or @konstin could weigh in?