From 5be1d17d5fc42c8e664e0382caf07915e5a0bd19 Mon Sep 17 00:00:00 2001 From: Grzegorz Bokota Date: Fri, 10 Feb 2023 19:56:24 +0000 Subject: [PATCH 01/10] add information about lack of possibility of cross compile Poetry projects --- docs/faq.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/faq.md b/docs/faq.md index ba0eed473..b690ee994 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -20,7 +20,13 @@ Linux wheels are built in [`manylinux`/`musllinux` containers](https://github.co ### Building macOS wheels for Apple Silicon {: #apple-silicon} -`cibuildwheel` supports cross-compiling `universal2` and `arm64` wheels on `x86_64` runners. With the introduction of Apple Silicon, you now have several choices for wheels for Python 3.8+: +`cibuildwheel` supports cross-compiling `universal2` and `arm64` wheels on `x86_64` runners. + +!!! note + Currently it is impossible to cross-compiling project that are using Poetry as a build backend. + Reladed issue [here](https://github.com/python-poetry/poetry/issues/7107) + +With the introduction of Apple Silicon, you now have several choices for wheels for Python 3.8+: #### `x86_64` From cc0bb5da34399c8e630aa69f7079dc30d8a11656 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Fri, 10 Feb 2023 20:00:17 +0000 Subject: [PATCH 02/10] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- docs/faq.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/faq.md b/docs/faq.md index b690ee994..3cb1034c3 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -20,10 +20,10 @@ Linux wheels are built in [`manylinux`/`musllinux` containers](https://github.co ### Building macOS wheels for Apple Silicon {: #apple-silicon} -`cibuildwheel` supports cross-compiling `universal2` and `arm64` wheels on `x86_64` runners. +`cibuildwheel` supports cross-compiling `universal2` and `arm64` wheels on `x86_64` runners. !!! note - Currently it is impossible to cross-compiling project that are using Poetry as a build backend. + Currently it is impossible to cross-compiling project that are using Poetry as a build backend. Reladed issue [here](https://github.com/python-poetry/poetry/issues/7107) With the introduction of Apple Silicon, you now have several choices for wheels for Python 3.8+: From 96cffc2c7c4a9160f355c43cbcd32e36e9c49694 Mon Sep 17 00:00:00 2001 From: Grzegorz Bokota Date: Fri, 10 Feb 2023 22:56:52 +0100 Subject: [PATCH 03/10] Update docs/faq.md Co-authored-by: Henry Schreiner --- docs/faq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/faq.md b/docs/faq.md index 3cb1034c3..073b3a94d 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -23,7 +23,7 @@ Linux wheels are built in [`manylinux`/`musllinux` containers](https://github.co `cibuildwheel` supports cross-compiling `universal2` and `arm64` wheels on `x86_64` runners. !!! note - Currently it is impossible to cross-compiling project that are using Poetry as a build backend. + Currently it is impossible to cross-compile project that uses Poetry as a build backend. Reladed issue [here](https://github.com/python-poetry/poetry/issues/7107) With the introduction of Apple Silicon, you now have several choices for wheels for Python 3.8+: From c7643f08e756dfd6ece75b106bd7950ea7bb3423 Mon Sep 17 00:00:00 2001 From: Grzegorz Bokota Date: Tue, 14 Feb 2023 13:31:20 +0000 Subject: [PATCH 04/10] update information about poetry state --- docs/faq.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/faq.md b/docs/faq.md index 073b3a94d..78bf8e19a 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -23,8 +23,13 @@ Linux wheels are built in [`manylinux`/`musllinux` containers](https://github.co `cibuildwheel` supports cross-compiling `universal2` and `arm64` wheels on `x86_64` runners. !!! note - Currently it is impossible to cross-compile project that uses Poetry as a build backend. - Reladed issue [here](https://github.com/python-poetry/poetry/issues/7107) + Currently it is not fully supported to cross-compile project that uses Poetry as a build backend. + Reladed issue [here](https://github.com/python-poetry/poetry/issues/7107). + Some packages can be built with arm64 wheels, but their file names will be incorrect, + with the platform tag showing `x86_64` instead of `arm64`. + As a workaround, the file can be renamed before running delocate to repair the wheel. + For further details, please refer to this (https://github.com/pypa/cibuildwheel/pull/1416#discussion_r1105217074)." + With the introduction of Apple Silicon, you now have several choices for wheels for Python 3.8+: From 567fe2a92316a862525320413f3db47476863621 Mon Sep 17 00:00:00 2001 From: Grzegorz Bokota Date: Wed, 15 Feb 2023 00:00:22 +0100 Subject: [PATCH 05/10] Apply suggestions from code review Co-authored-by: Henry Schreiner --- docs/faq.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/faq.md b/docs/faq.md index 78bf8e19a..aea2726dc 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -24,8 +24,8 @@ Linux wheels are built in [`manylinux`/`musllinux` containers](https://github.co !!! note Currently it is not fully supported to cross-compile project that uses Poetry as a build backend. - Reladed issue [here](https://github.com/python-poetry/poetry/issues/7107). - Some packages can be built with arm64 wheels, but their file names will be incorrect, + Related issue [here](https://github.com/python-poetry/poetry/issues/7107). + Some packages can be built with arm64 wheels, but their tags will be incorrect, with the platform tag showing `x86_64` instead of `arm64`. As a workaround, the file can be renamed before running delocate to repair the wheel. For further details, please refer to this (https://github.com/pypa/cibuildwheel/pull/1416#discussion_r1105217074)." From ed320412b0ff31ff4111ee175c8b431894cbbd3f Mon Sep 17 00:00:00 2001 From: Grzegorz Bokota Date: Thu, 16 Feb 2023 18:20:37 +0000 Subject: [PATCH 06/10] fix note text --- docs/faq.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/faq.md b/docs/faq.md index aea2726dc..108ac97a8 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -27,8 +27,7 @@ Linux wheels are built in [`manylinux`/`musllinux` containers](https://github.co Related issue [here](https://github.com/python-poetry/poetry/issues/7107). Some packages can be built with arm64 wheels, but their tags will be incorrect, with the platform tag showing `x86_64` instead of `arm64`. - As a workaround, the file can be renamed before running delocate to repair the wheel. - For further details, please refer to this (https://github.com/pypa/cibuildwheel/pull/1416#discussion_r1105217074)." + As a workaround, the tag should be fixed before running delocate to repair the wheel. With the introduction of Apple Silicon, you now have several choices for wheels for Python 3.8+: From 983347f370b43d5a55db6332a155ef85e9adad44 Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Sun, 18 Jun 2023 17:49:57 +0100 Subject: [PATCH 07/10] Add reference to the new 'wheel tags' command --- docs/faq.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/faq.md b/docs/faq.md index 108ac97a8..98961820a 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -23,11 +23,11 @@ Linux wheels are built in [`manylinux`/`musllinux` containers](https://github.co `cibuildwheel` supports cross-compiling `universal2` and `arm64` wheels on `x86_64` runners. !!! note - Currently it is not fully supported to cross-compile project that uses Poetry as a build backend. - Related issue [here](https://github.com/python-poetry/poetry/issues/7107). - Some packages can be built with arm64 wheels, but their tags will be incorrect, - with the platform tag showing `x86_64` instead of `arm64`. - As a workaround, the tag should be fixed before running delocate to repair the wheel. + Currently it is not fully supported to cross-compile project that uses **Poetry** as a build backend. Related issue [here](https://github.com/python-poetry/poetry/issues/7107). + + Some packages can be built with arm64 wheels, but their tags will be incorrect, with the platform tag showing `x86_64` instead of `arm64`. + + As a workaround, the tag should be fixed before running delocate to repair the wheel. The [`wheel tags`](https://wheel.readthedocs.io/en/stable/reference/wheel_tags.html) command is ideal for this. See [this workflow](https://gist.github.com/anderssonjohan/49f07e33fc5cb2420515a8ac76dc0c95#file-build-pendulum-wheels-yml-L39-L53) for an example usage. With the introduction of Apple Silicon, you now have several choices for wheels for Python 3.8+: From 3192cdcbfa0a3aefe661bd4bee1669aa65ac9875 Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Sun, 18 Jun 2023 18:03:06 +0100 Subject: [PATCH 08/10] Copy pass on this text --- docs/faq.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/docs/faq.md b/docs/faq.md index 98961820a..ec4e6b7da 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -23,12 +23,9 @@ Linux wheels are built in [`manylinux`/`musllinux` containers](https://github.co `cibuildwheel` supports cross-compiling `universal2` and `arm64` wheels on `x86_64` runners. !!! note - Currently it is not fully supported to cross-compile project that uses **Poetry** as a build backend. Related issue [here](https://github.com/python-poetry/poetry/issues/7107). - - Some packages can be built with arm64 wheels, but their tags will be incorrect, with the platform tag showing `x86_64` instead of `arm64`. - - As a workaround, the tag should be fixed before running delocate to repair the wheel. The [`wheel tags`](https://wheel.readthedocs.io/en/stable/reference/wheel_tags.html) command is ideal for this. See [this workflow](https://gist.github.com/anderssonjohan/49f07e33fc5cb2420515a8ac76dc0c95#file-build-pendulum-wheels-yml-L39-L53) for an example usage. + If your project uses **Poetry** as a build backend, cross-compiling on macOS [does not currently work](https://github.com/python-poetry/poetry/issues/7107). In some cases arm64 wheels can be built but their tags will be incorrect, with the platform tag showing `x86_64` instead of `arm64`. + As a workaround, the tag can be fixed before running delocate to repair the wheel. The [`wheel tags`](https://wheel.readthedocs.io/en/stable/reference/wheel_tags.html) command is ideal for this. See [this workflow](https://gist.github.com/anderssonjohan/49f07e33fc5cb2420515a8ac76dc0c95#file-build-pendulum-wheels-yml-L39-L53) for an example usage of `wheel tags`. With the introduction of Apple Silicon, you now have several choices for wheels for Python 3.8+: From 1ce209198aabbb3fa0a96c3ab799c1077da9ddb1 Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Sun, 18 Jun 2023 19:12:29 +0100 Subject: [PATCH 09/10] Copy edit of the whole section, make it a bit easier to follow, add some details about universal2 --- docs/faq.md | 55 +++++++++++++++++++++++++++++++---------------------- 1 file changed, 32 insertions(+), 23 deletions(-) diff --git a/docs/faq.md b/docs/faq.md index ec4e6b7da..f7256b89c 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -22,14 +22,11 @@ Linux wheels are built in [`manylinux`/`musllinux` containers](https://github.co `cibuildwheel` supports cross-compiling `universal2` and `arm64` wheels on `x86_64` runners. -!!! note - If your project uses **Poetry** as a build backend, cross-compiling on macOS [does not currently work](https://github.com/python-poetry/poetry/issues/7107). In some cases arm64 wheels can be built but their tags will be incorrect, with the platform tag showing `x86_64` instead of `arm64`. - - As a workaround, the tag can be fixed before running delocate to repair the wheel. The [`wheel tags`](https://wheel.readthedocs.io/en/stable/reference/wheel_tags.html) command is ideal for this. See [this workflow](https://gist.github.com/anderssonjohan/49f07e33fc5cb2420515a8ac76dc0c95#file-build-pendulum-wheels-yml-L39-L53) for an example usage of `wheel tags`. +#### Overview of Mac architectures With the introduction of Apple Silicon, you now have several choices for wheels for Python 3.8+: -#### `x86_64` +##### `x86_64` The traditional wheel for Apple, loads on Intel machines, and on Apple Silicon when running Python under Rosetta 2 emulation. @@ -37,51 +34,63 @@ Apple Silicon when running Python under Rosetta 2 emulation. Due to a change in naming, Pip 20.3+ (or an installer using packaging 20.5+) is required to install a binary wheel on macOS Big Sur. -#### `arm64` +##### `arm64` The native wheel for macOS on Apple Silicon. Requires Pip 20.3+ (or packaging 20.5+) to install. -#### `universal2` +##### `universal2` This wheel contains both architectures, causing it to be up to twice the size (data files do not get doubled, only compiled code). It requires Pip 20.3 (Packaging 20.6+) to load on Intel, and Pip 21.0.1 (Packaging 20.9+) to load on Apple Silicon. -!!! note - The dual-architecture `universal2` has a few benefits, but a key benefit - to a universal wheel is that a user can bundle these wheels into an - application and ship a single binary. +The dual-architecture `universal2` has a few benefits, but a key benefit +to a universal wheel is that a user can bundle these wheels into an +application and ship a single binary. + +However, if you have a large library, then you might prefer to ship +the two single-arch wheels instead - `x86_64` and `arm64`. In rare cases, +you might want to build all three, but in that case, pip will not download +the universal wheels, because it prefers the most specific wheel +available. - However, if you have a large library, then you might prefer to ship - the two single-arch wheels instead - `x86_64` and `arm64`. In rare cases, - you might want to build all three, but in that case, pip will not download - the universal wheels, because it prefers the most specific wheel - available. +#### What to provide? Generally speaking, because Pip 20.3 is required for the `universal2` wheel, most packages should provide both `x86_64` and one of `universal2`/`arm64` -wheels for now. When Pip 20.3+ is common on macOS, then it might be possible -to ship only the `universal2` wheel. +wheels. When Pip 20.3+ is common on macOS, then it might be possible to ship +only the `universal2` wheel. -**Apple Silicon wheels are not built by default on Intel runners**, but can be enabled by adding extra archs to the [`CIBW_ARCHS_MACOS` option](options.md#archs) - e.g. `x86_64 arm64`. Cross-compilation is provided by the Xcode toolchain. +Opinions vary on which of arch-specific or `universal2` wheels are best - some packagers prefer `universal2` because it's one wheel for all Mac users, so simpler, and easier to build into apps for downstream users. However, because they contain code for both architectures, their file size is larger, meaning they consume more disk space and bandwidth, and are harder to build for some projects. + +See [GitHub issue 1333](https://github.com/pypa/cibuildwheel/issues/1333) for more discussion. + +#### How to cross-compile + +With the exception of Cirrus CI, macOS CI runners are still Intel-based, and Apple Silicon wheels are not built by default on Intel runners. However, cross-compilation can be enabled by adding extra archs to the [`CIBW_ARCHS_MACOS` option](options.md#archs) - e.g. `x86_64 arm64`. Cross-compilation is provided by the Xcode toolchain. !!! important When cross-compiling on Intel, it is not possible to test `arm64` and the `arm64` part of a `universal2` wheel. `cibuildwheel` will raise a warning to notify you of this - these warnings be be silenced by skipping testing on these platforms: `CIBW_TEST_SKIP: *_arm64 *_universal2:arm64`. -Hopefully, cross-compilation is a temporary situation. Once we have widely -available Apple Silicon CI runners, we can build and test `arm64` and -`universal2` wheels natively. That's why `universal2`/`arm64` wheels require opt-in by setting `CIBW_ARCHS_MACOS`. - !!! note Your runner needs Xcode Command Line Tools 12.2 or later to build `universal2` or `arm64`. Only CPython 3.8 and newer support `universal2` and `arm64` wheels. +!!! note + If your project uses **Poetry** as a build backend, cross-compiling on macOS [does not currently work](https://github.com/python-poetry/poetry/issues/7107). In some cases arm64 wheels can be built but their tags will be incorrect, with the platform tag showing `x86_64` instead of `arm64`. + + As a workaround, the tag can be fixed before running delocate to repair the wheel. The [`wheel tags`](https://wheel.readthedocs.io/en/stable/reference/wheel_tags.html) command is ideal for this. See [this workflow](https://gist.github.com/anderssonjohan/49f07e33fc5cb2420515a8ac76dc0c95#file-build-pendulum-wheels-yml-L39-L53) for an example usage of `wheel tags`. + +Hopefully, cross-compilation is a temporary situation. Once we have widely +available Apple Silicon CI runners, we can build and test `arm64` and +`universal2` wheels natively. That's why `universal2`/`arm64` wheels require opt-in by setting `CIBW_ARCHS_MACOS`. + Here's an example GitHub Actions workflow with a job that builds for Apple Silicon: > .github/workflows/build_macos.yml From f44d2784f85a87239000a6be8661f40fc908b497 Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Sun, 9 Jul 2023 15:17:51 +0100 Subject: [PATCH 10/10] Update docs/faq.md --- docs/faq.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/faq.md b/docs/faq.md index f3497474e..035bc4edf 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -87,9 +87,7 @@ With the exception of Cirrus CI, macOS CI runners are still Intel-based, and App As a workaround, the tag can be fixed before running delocate to repair the wheel. The [`wheel tags`](https://wheel.readthedocs.io/en/stable/reference/wheel_tags.html) command is ideal for this. See [this workflow](https://gist.github.com/anderssonjohan/49f07e33fc5cb2420515a8ac76dc0c95#file-build-pendulum-wheels-yml-L39-L53) for an example usage of `wheel tags`. -Hopefully, cross-compilation is a temporary situation. Once we have widely -available Apple Silicon CI runners, we can build and test `arm64` and -`universal2` wheels natively. That's why `universal2`/`arm64` wheels require opt-in by setting `CIBW_ARCHS_MACOS`. +Hopefully, cross-compilation is a temporary situation. Once we have widely available Apple Silicon CI runners, we can build `arm64` wheels natively. Until then, cross-compiling `arm64` or `universal2` wheels requires opt-in by setting `CIBW_ARCHS_MACOS` on Intel runners. Here's an example GitHub Actions workflow with a job that builds for Apple Silicon: