cross-compilation support

[Gankra]

This is punted on in the current version, pending some evaluation of the options. There are three four options I know:

• cross-rs (established, based on docker)
• cargo-zigbuild (new kid on the block, has some blocking issues)
• cargo-xwin (very specific to targetting windows? just learned about this.)
• Do It Our Dang Selves With Rustup (please god no)

This is a blocker for musl and aarch64 (arm64) support.

Note that taiki-e/upload-rust-binary-action does handle this and in my experiments whatever it does Just Works (great work from taiki-e as always).

[Gankra]

One of my big concerns about cross-compilation is to what extent certain Fancy Things cannot be cross-compiled.

Apple certainly does not want you cross-compiling from linux to macos. I'm not sure if support for code signing (#21) or certain installers is necessarily non-crossable. Certainly some are harder to cross. Certainly there are... Ways Around The Rules. But I don't want to get people in trouble or have them ship busted things!

[reubenmiller]

Just some of my recent experience/pain regarding cross compiling for Linux arm* and x86_64 targets for an open source project that I'm involved with.

Basically we have to build for linux environments on arm >= v6, arm64 and x86_64, for both gnu and musl releases. We were using cargo-cross however still had some issues to build for all the targets and the output was dependant on your build machine.

Then I saw a great blog post about cargo-zigbuild and was amazed with how effort it was to cross compile for all of our targets, especially since the other developers have a mix between Windows (using WSL2) and MacOS (M1/arm64 and amd64) setups.

Below details some of the details with the projects:

Experience cargo-cross

• Slow builds as building on MacOS M1 (arm64/aarch64) and the current default is to use amd64 images by default (though there are unofficial arm64 available...there is also a "zigbuild" variant, from what I found digging around)
• Building for musl on some targets just didn't work (mostly because of our project's dependency to the common ring...the number of times that dependency has caused me build headaches is just astounding)

Experience with cargo-zigbuild

• Builds are quick
• Does not rely on docker
• Feels more like rust, as you just use cargo target add xxx, then cargo zigbuild --target xxx
• Dependencies are included (ok, still had to install ziglang via pip first but that was it)

Overall I was finally able to get the following targets built using cargo-zigbuild running both natively on MacOS M1 and inside a linux (dev) container.

• x86_64-unknown-linux-musl
• aarch64-unknown-linux-musl
• armv7-unknown-linux-musleabihf
• arm-unknown-linux-musleabihf
• x86_64-unknown-linux-gnu
• aarch64-unknown-linux-gnu
• armv7-unknown-linux-gnueabihf (yes, this does actually work...I think notice in the project's README.md is out of date)
• arm-unknown-linux-gnueabihf

Though like you said, I don't think it will work for absolutely everything, so it might be worthwhile allowing some "composability" when it comes to building for different targets. And the targets are above are not so exotic compared to the other more lower level embedded targets, e.g. risc etc.

I created a gist to detail my experience, including the references to sources etc.

I hope my experience can at least help someone else, as this had caused me countless hours fluffy around with various dependencies/build machines.

[messense]

• armv7-unknown-linux-gnueabihf (yes, this does actually work...I think notice in the project's README.md is out of date)

To clarify, I have implemented some workarounds for these caveats in cargo-zigbuild and updated the readme: https://github.com/rust-cross/cargo-zigbuild#caveats

[VorpalBlade]

Is there any work being done on this and/or is there any way to help on this? Cross compiling to Linux ARM and Linux Aarch64 is an absolute must for me.

[VorpalBlade]

@reubenmiller I found the opposite. Zigbuild is great when it works, but:

1. It fails when trying to for example cross compile and link OpenSSL (openssl-sys crate). You can either enabled the vendored feature of openssl-sys to build OpenSSL from source or with cross install the relevant -dev package for cross compiling in a pre-build step in the docker container.
   Installing OpenSSL as a pre-build step is much faster than building it from scratch every time, easily negating the general speed improvement of zigbuild over cross. (And no, rustls/ring is not an option for several of the platforms I'm targeting.)
2. Cross supports running the tests using user mode qemu, zigbuild doesn't as far as I have been able to find. While this is not important for release, it is important for CI in general. And I'd rather not have to maintain two separate cross compilation systems.
3. Cross compiling to Windows (x86_64-pc-windows-gnu) from Linux just doesn't seem to work at at all with zigbuild, works out of the box with cross.

Thus I think cross is the better and more mature option to go with.

[cryingpotat0]

👋 Hey! I also wanted to chime in and see if cross compilation for cost reduction purposes was on the roadmap? Spinning up separate machines (especially expensive Mac machines, almost 10x an equivalent Linux machine) for every release seems like it could blow past the free tier on Github actions. While it's still cheap in the grand scheme of things, it would be cool to use the cheapest Github actions instance to compile for all platforms where possible.

[ssokolow]

Unfortunately, mac machines are probably the most difficult to replace because, unlike with MinGW, there's no libre equivalent to some of the parts of Apple's toolchain. (I believe the binary-signing tool is one such example, but that may be outdated information.)

The solutions I've run into which work without mac hardware have involved using parts of Apple's toolchain on top of non-mac hardware in violation of its license.