Test cibuildwheel for macOS wheels #232
Conversation
Try out a basic cibuildwheel workflow just for macOS wheels for now. The main aim is to get arm64 wheels for Apple Silicon (M1) processors. On success this could be extended to manylinux wheels too...
|
That would be very welcome! You probably noticed, but the address of the measures data is ftp://ftp.astron.nl/outgoing/Measures/WSRT_Measures.ztar |
Compiling the python package does not depend on the exact version of the measures data. The latest is fine. This avoids crashing the build if casacore-data is out of date, which happens from time to time. Since ftp.astron.nl only stores the latest 3 months or so, this requires casacore-data to be updated every 3 months in sync with it, and it doesn't always happen.
|
Yes, just discovered that 🙂 I had hoped that this pull request would also exercise the new workflow but it didn't... I'm restarting my effort on my own fork in master branch to test this, then I'll add some more commits to this branch (or PR the fork). |
|
The one issue is boost (as always!). I'll check if there's a way to make multiple pythons with their corresponding boosts without compiling casacore 5-15 times :-) |
|
I'm gaining more experience with pybind11 in other projects. Porting python-casacore to pybind11 is (still) on my todo-list, but I'm getting more optimistic that it's easy. |
|
Yes, long-term that's probably the best solution. |
|
Unfortunately this looks like a dead end for M1 wheels... for now. I've successfully built arm64 wheels for python-gnureadline using cibuildwheel, but there I'm compiling readline myself as a universal binary. The wheel machinery then strips the dylib appropriately to produce The problem here is that we also need to link in the casacore dylibs. Since the macOS runner uses Homebrew and is intel only at this point, you'll get single-arch x86_64 casacore. (Homebrew won't support universal builds anymore.) Once we have M1 runners in GitHub Actions we'll be sorted (at least for single-arch wheels), but that could still take a while. I should have an M1 laptop soon, and then I'll experiment with direct installation of this package and producing some arm64 wheels to help out. I could also potentially offer the laptop as a remote runner to go this route :-) Leaving the PR open for now to investigate the latter option soon. |
|
@ludwigschwardt Once your laptop arrives, you could try the newest M1 wheel for python-casacore 3.5. |
This is based on the latest github-minimal.yml example in cibuildwheel. Enable tests on arm64 again, since there should be M1 runners now. Drop the universal build to save time and space, since Homebrew doesn't dig it either.
The Actions tab is complaining about v2 that uses the deprecated Node12 and this causes the tests to fail. See the following post for more details: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
Unfortunately the manual dispatch button only works on the default (master) branch, so we need to trigger via pushing.
Python-casacore CMake fails with: Could NOT find Python3 (missing: Python3_NumPy_INCLUDE_DIRS NumPy) This is similar to https://issues.apache.org/jira/browse/ARROW-11151, where a workaround is to install numpy first (even though Homebrew also installs it).
This is necessary since we otherwise have to install multiple boost-python3's, one per minor Python version in cibuildwheel. It is sort-of possible (see https://github.com/MarkusJx/install-boost) but with some restrictions (no Apple Silicon?) Also restore unit test restrictions, since GitHub still does not have M1 runner support (see github/roadmap#528).
|
Update: The one sad part is that I was waiting for M1 GitHub runners to simplify building and testing. They have finally arrived in October but you have to pay for them 🤦🏼♂️ As it stands, you need to build M1 wheels on an M1, because you have to link against libcasacore and if you use Homebrew for that you won't get a universal binary but one which reflects the runner's environment. I was thinking of building those wheels on my laptop for starters. Alternatively we need to build libcasacore outside Homebrew. |
I'm trying out a basic cibuildwheel workflow just for macOS wheels for now.
The main aim is to get those sweet arm64 wheels for Apple Silicon (M1) processors. 😁
On success this could be extended to manylinux wheels too...