Cross-compiling for x86_64-unknown-freebsd from macOS fails

[kellytk]

macOS High Sierra (10.13.4)

$ cargo build --target x86_64-unknown-freebsd
error: failed to run custom build command for `backtrace-sys v0.1.22`
process didn't exit successfully: `/Users/user/project/target/debug/build/backtrace-sys-59d4ec60a7871227/build-script-build` (exit code: 101)

--- stdout
TARGET = Some("x86_64-unknown-freebsd")
OPT_LEVEL = Some("0")
TARGET = Some("x86_64-unknown-freebsd")
HOST = Some("x86_64-apple-darwin")
TARGET = Some("x86_64-unknown-freebsd")
TARGET = Some("x86_64-unknown-freebsd")
HOST = Some("x86_64-apple-darwin")
CC_x86_64-unknown-freebsd = None
CC_x86_64_unknown_freebsd = None
TARGET_CC = None
CC = None
HOST = Some("x86_64-apple-darwin")
CROSS_COMPILE = None
TARGET = Some("x86_64-unknown-freebsd")
HOST = Some("x86_64-apple-darwin")
CFLAGS_x86_64-unknown-freebsd = None
CFLAGS_x86_64_unknown_freebsd = None
TARGET_CFLAGS = None
CFLAGS = None
DEBUG = Some("true")
running: "cc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-m64" "-I" "src/libbacktrace" "-I" "/Users/user/project/target/x86_64-unknown-freebsd/debug/build/backtrace-sys-6eb4eda84a25fa8a/out" "-fvisibility=hidden" "-DBACKTRACE_ELF_SIZE=64" "-DBACKTRACE_SUPPORTED=1" "-DBACKTRACE_USES_MALLOC=1" "-DBACKTRACE_SUPPORTS_THREADS=0" "-DBACKTRACE_SUPPORTS_DATA=0" "-DHAVE_DL_ITERATE_PHDR=1" "-D_GNU_SOURCE=1" "-D_LARGE_FILES=1" "-Dbacktrace_full=__rbt_backtrace_full" "-Dbacktrace_dwarf_add=__rbt_backtrace_dwarf_add" "-Dbacktrace_initialize=__rbt_backtrace_initialize" "-Dbacktrace_pcinfo=__rbt_backtrace_pcinfo" "-Dbacktrace_syminfo=__rbt_backtrace_syminfo" "-Dbacktrace_get_view=__rbt_backtrace_get_view" "-Dbacktrace_release_view=__rbt_backtrace_release_view" "-Dbacktrace_alloc=__rbt_backtrace_alloc" "-Dbacktrace_free=__rbt_backtrace_free" "-Dbacktrace_vector_finish=__rbt_backtrace_vector_finish" "-Dbacktrace_vector_grow=__rbt_backtrace_vector_grow" "-Dbacktrace_vector_release=__rbt_backtrace_vector_release" "-Dbacktrace_close=__rbt_backtrace_close" "-Dbacktrace_open=__rbt_backtrace_open" "-Dbacktrace_print=__rbt_backtrace_print" "-Dbacktrace_simple=__rbt_backtrace_simple" "-Dbacktrace_qsort=__rbt_backtrace_qsort" "-Dbacktrace_create_state=__rbt_backtrace_create_state" "-Dbacktrace_uncompress_zdebug=__rbt_backtrace_uncompress_zdebug" "-o" "/Users/user/project/target/x86_64-unknown-freebsd/debug/build/backtrace-sys-6eb4eda84a25fa8a/out/src/libbacktrace/elf.o" "-c" "src/libbacktrace/elf.c"
cargo:warning=src/libbacktrace/elf.c:43:10: fatal error: 'link.h' file not found
cargo:warning=#include <link.h>
cargo:warning=         ^~~~~~~~
cargo:warning=1 error generated.
exit code: 1

--- stderr
thread 'main' panicked at '

Internal error occurred: Command "cc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-m64" "-I" "src/libbacktrace" "-I" "/Users/user/project/target/x86_64-unknown-freebsd/debug/build/backtrace-sys-6eb4eda84a25fa8a/out" "-fvisibility=hidden" "-DBACKTRACE_ELF_SIZE=64" "-DBACKTRACE_SUPPORTED=1" "-DBACKTRACE_USES_MALLOC=1" "-DBACKTRACE_SUPPORTS_THREADS=0" "-DBACKTRACE_SUPPORTS_DATA=0" "-DHAVE_DL_ITERATE_PHDR=1" "-D_GNU_SOURCE=1" "-D_LARGE_FILES=1" "-Dbacktrace_full=__rbt_backtrace_full" "-Dbacktrace_dwarf_add=__rbt_backtrace_dwarf_add" "-Dbacktrace_initialize=__rbt_backtrace_initialize" "-Dbacktrace_pcinfo=__rbt_backtrace_pcinfo" "-Dbacktrace_syminfo=__rbt_backtrace_syminfo" "-Dbacktrace_get_view=__rbt_backtrace_get_view" "-Dbacktrace_release_view=__rbt_backtrace_release_view" "-Dbacktrace_alloc=__rbt_backtrace_alloc" "-Dbacktrace_free=__rbt_backtrace_free" "-Dbacktrace_vector_finish=__rbt_backtrace_vector_finish" "-Dbacktrace_vector_grow=__rbt_backtrace_vector_grow" "-Dbacktrace_vector_release=__rbt_backtrace_vector_release" "-Dbacktrace_close=__rbt_backtrace_close" "-Dbacktrace_open=__rbt_backtrace_open" "-Dbacktrace_print=__rbt_backtrace_print" "-Dbacktrace_simple=__rbt_backtrace_simple" "-Dbacktrace_qsort=__rbt_backtrace_qsort" "-Dbacktrace_create_state=__rbt_backtrace_create_state" "-Dbacktrace_uncompress_zdebug=__rbt_backtrace_uncompress_zdebug" "-o" "/Users/user/project/target/x86_64-unknown-freebsd/debug/build/backtrace-sys-6eb4eda84a25fa8a/out/src/libbacktrace/elf.o" "-c" "src/libbacktrace/elf.c" with args "cc" did not execute successfully (status code exit code: 1).

', /Users/user/.cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.15/src/lib.rs:2158:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.

warning: build failed, waiting for other jobs to finish...
error: build failed

[alexcrichton]

Cross compilation isn't natively supported so this is currently an expected error, you'll need to install a toolchain that can compile code for FreeBSD

[Timmmm]

I got the same error cross-compiling to Linux from Mac:

"/usr/local/opt/ccache/libexec/clang" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=x86_64-unknown-linux-musl" "-I" "src/libbacktrace" "-I" "/Users/me/backend/target/x86_64-unknown-linux-musl/release/build/backtrace-sys-dfaa951a1a3674cc/out" "-fvisibility=hidden" "-DBACKTRACE_ELF_SIZE=64" "-DBACKTRACE_SUPPORTED=1" "-DBACKTRACE_USES_MALLOC=1" "-DBACKTRACE_SUPPORTS_THREADS=0" "-DBACKTRACE_SUPPORTS_DATA=0" "-DHAVE_DL_ITERATE_PHDR=1" "-D_GNU_SOURCE=1" "-D_LARGE_FILES=1" "-Dbacktrace_full=__rbt_backtrace_full" "-Dbacktrace_dwarf_add=__rbt_backtrace_dwarf_add" "-Dbacktrace_initialize=__rbt_backtrace_initialize" "-Dbacktrace_pcinfo=__rbt_backtrace_pcinfo" "-Dbacktrace_syminfo=__rbt_backtrace_syminfo" "-Dbacktrace_get_view=__rbt_backtrace_get_view" "-Dbacktrace_release_view=__rbt_backtrace_release_view" "-Dbacktrace_alloc=__rbt_backtrace_alloc" "-Dbacktrace_free=__rbt_backtrace_free" "-Dbacktrace_vector_finish=__rbt_backtrace_vector_finish" "-Dbacktrace_vector_grow=__rbt_backtrace_vector_grow" "-Dbacktrace_vector_release=__rbt_backtrace_vector_release" "-Dbacktrace_close=__rbt_backtrace_close" "-Dbacktrace_open=__rbt_backtrace_open" "-Dbacktrace_print=__rbt_backtrace_print" "-Dbacktrace_simple=__rbt_backtrace_simple" "-Dbacktrace_qsort=__rbt_backtrace_qsort" "-Dbacktrace_create_state=__rbt_backtrace_create_state" "-Dbacktrace_uncompress_zdebug=__rbt_backtrace_uncompress_zdebug" "-Dmacho_get_view=__rbt_macho_get_view" "-Dmacho_symbol_type_relevant=__rbt_macho_symbol_type_relevant" "-Dmacho_get_commands=__rbt_macho_get_commands" "-Dmacho_try_dsym=__rbt_macho_try_dsym" "-Dmacho_try_dwarf=__rbt_macho_try_dwarf" "-Dmacho_get_addr_range=__rbt_macho_get_addr_range" "-Dmacho_get_uuid=__rbt_macho_get_uuid" "-Dmacho_add=__rbt_macho_add" "-Dmacho_add_symtab=__rbt_macho_add_symtab" "-Dmacho_file_to_host_u64=__rbt_macho_file_to_host_u64" "-Dmacho_file_to_host_u32=__rbt_macho_file_to_host_u32" "-Dmacho_file_to_host_u16=__rbt_macho_file_to_host_u16" "-o" "/Users/me/backend/target/x86_64-unknown-linux-musl/release/build/backtrace-sys-dfaa951a1a3674cc/out/src/libbacktrace/elf.o" "-c" "src/libbacktrace/elf.c"

My understanding was that you could use normal Clang to cross-compile code to any supported target using the --target option, but you still have to use a special cross-compiling linker. As I understand it, ATOM LLD is an attempt to make a sane linker that can target any platform, like Clang itself can, but it only supports Mac so far.

So I installed this brew formula to get a cross-compiling linker and it works perfectly for pure Rust code.

It seems like the problem is that Cargo doesn't tell Clang the include path of MUSL? I have link.h here:

/usr/local/Cellar/musl-cross/0.9.9/libexec/x86_64-linux-musl/include/link.h

But how do I add -I/usr/local/Cellar/musl-cross/0.9.9/libexec/x86_64-linux-musl/include to the above command line?

[Timmmm]

Ooo I figured it out. Set the CFLAGS environment variable. Here's my Python code to run it:

        cargo_env = os.environ.copy()
        cargo_env["CARGO_TARGET_x86_64-unknown-linux-musl_LINKER"] = "x86_64-linux-musl-gcc"
        cargo_env["CFLAGS"] = "-I/usr/local/Cellar/musl-cross/0.9.9/libexec/x86_64-linux-musl/include"
        subprocess.run(
          [
            "cargo",
            "build",
            "--target",
            "x86_64-unknown-linux-musl",
          ] + build_arg,
          cwd="backend",
          check=True,
          env=cargo_env,
        )

My build still fails on the final link but it gets further at least!

[Timmmm]

Ah got it all working! Turns out the target name has to be converted to uppercase and underscores:

        cargo_env["CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER"] = "x86_64-linux-musl-gcc"

It's mentioned in the documentation here but not actually in the list of environment variables here unfortunately.

[Timmmm]

Made a pull request to clarify the triple case: rust-lang/cargo#8079