How to use direct cross-compilation under Linux

[FengZh61]

I want to compile the executable file of aarch64-unknown-linux-gnu under x86_64 GNU/Linux. How to configure the project parameters is needed. I have installed gcc-aarch64-linux-gnu \ g++-aarch64-linux-gnu \binutils-aarch64-linux-gnu \libc6-dev-arm64-cross.

Using cross-compilation ultimately depends on the glibc version of the compilation environment, which may be higher than my running environment.

error: linking with `aarch64-linux-gnu-gcc` failed: exit status: 1
  |
  = note:  "aarch64-linux-gnu-gcc" "<1 object files omitted>" "-Wl,--as-needed" "-Wl,-Bstatic" "/opt/vector/target/aarch64-unknown-linux-gnu/release/deps/rustcBi5hre/{libtikv_jemalloc_sys-466b46d7cba71470,liblz4_sys-b9c951accf8f125a,librdkafka_sys-80e4439d10555fb9,libcurl_sys-c0f5e59d0e4b0b35,liblibz_sys-4180caf8dc02cad4,libzstd_sys-fa40d6e9b3bad4e2,libring-4396bea31710f768,libopenssl_sys-60ca4bfdfd489a5c,libonig_sys-b3298023a2580ec9}.rlib" "-llua5.4" "<sysroot>/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcompiler_builtins-*.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-L" "/opt/vector/target/aarch64-unknown-linux-gnu/release/deps/rustcBi5hre/raw-dylibs" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/opt/vector/target/aarch64-unknown-linux-gnu/release/build/zstd-sys-291de93a985572e6/out" "-L" "/opt/vector/target/aarch64-unknown-linux-gnu/release/build/ring-2185f64283abbabf/out" "-L" "/opt/vector/target/aarch64-unknown-linux-gnu/release/build/openssl-sys-57e386abf17434f7/out/openssl-build/install/lib" "-L" "/opt/vector/target/aarch64-unknown-linux-gnu/release/build/onig_sys-b25a1dfdf50cd694/out" "-L" "/opt/vector/target/aarch64-unknown-linux-gnu/release/build/mlua-sys-7233e320124d7208/out/lib" "-L" "/opt/vector/target/aarch64-unknown-linux-gnu/release/build/lz4-sys-dfaf06ae2df5afe1/out" "-L" "/opt/vector/target/aarch64-unknown-linux-gnu/release/build/rdkafka-sys-0da974230f159343/out/src" "-L" "/opt/vector/target/aarch64-unknown-linux-gnu/release/build/curl-sys-694423d109060bec/out/build" "-L" "/opt/vector/target/aarch64-unknown-linux-gnu/release/build/libz-sys-95c9254046279325/out/lib" "-L" "/opt/vector/target/aarch64-unknown-linux-gnu/release/build/tikv-jemalloc-sys-25bd0b78d0e824b5/out/lib" "-L" "<sysroot>/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-o" "/opt/vector/target/aarch64-unknown-linux-gnu/release/deps/vector-6e5e892360590c87" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-Wl,--strip-debug" "-nodefaultlibs" "-rdynamic"
  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: /usr/lib/gcc-cross/aarch64-linux-gnu/8/../../../../aarch64-linux-gnu/bin/ld: /opt/vector/target/aarch64-unknown-linux-gnu/release/deps/rustcBi5hre/librdkafka_sys-80e4439d10555fb9.rlib(rdkafka.o): Relocations in generic ELF (EM: 62)
          /usr/lib/gcc-cross/aarch64-linux-gnu/8/../../../../aarch64-linux-gnu/bin/ld: /opt/vector/target/aarch64-unknown-linux-gnu/release/deps/rustcBi5hre/librdkafka_sys-80e4439d10555fb9.rlib(rdkafka.o): Relocations in generic ELF (EM: 62)
          /usr/lib/gcc-cross/aarch64-linux-gnu/8/../../../../aarch64-linux-gnu/bin/ld: /opt/vector/target/aarch64-unknown-linux-gnu/release/deps/rustcBi5hre/librdkafka_sys-80e4439d10555fb9.rlib(rdkafka.o): Relocations in generic ELF (EM: 62)
          /usr/lib/gcc-cross/aarch64-linux-gnu/8/../../../../aarch64-linux-gnu/bin/ld: /opt/vector/target/aarch64-unknown-linux-gnu/release/deps/rustcBi5hre/librdkafka_sys-80e4439d10555fb9.rlib(rdkafka.o): Relocations in generic ELF (EM: 62)
          /usr/lib/gcc-cross/aarch64-linux-gnu/8/../../../../aarch64-linux-gnu/bin/ld: /opt/vector/target/aarch64-unknown-linux-gnu/release/deps/rustcBi5hre/librdkafka_sys-80e4439d10555fb9.rlib(rdkafka.o): Relocations in generic ELF (EM: 62)
          /usr/lib/gcc-cross/aarch64-linux-gnu/8/../../../../aarch64-linux-gnu/bin/ld: /opt/vector/target/aarch64-unknown-linux-gnu/release/deps/rustcBi5hre/librdkafka_sys-80e4439d10555fb9.rlib(rdkafka.o): Relocations in generic ELF (EM: 62)
          /usr/lib/gcc-cross/aarch64-linux-gnu/8/../../../../aarch64-linux-gnu/bin/ld: /opt/vector/target/aarch64-unknown-linux-gnu/release/deps/rustcBi5hre/librdkafka_sys-80e4439d10555fb9.rlib(rdkafka.o): Relocations in generic ELF (EM: 62)
          /usr/lib/gcc-cross/aarch64-linux-gnu/8/../../../../aarch64-linux-gnu/bin/ld: /opt/vector/target/aarch64-unknown-linux-gnu/release/deps/rustcBi5hre/librdkafka_sys-80e4439d10555fb9.rlib(rdkafka.o): Relocations in generic ELF (EM: 62)
          /usr/lib/gcc-cross/aarch64-linux-gnu/8/../../../../aarch64-linux-gnu/bin/ld: /opt/vector/target/aarch64-unknown-linux-gnu/release/deps/rustcBi5hre/librdkafka_sys-80e4439d10555fb9.rlib(rdkafka.o): Relocations in generic ELF (EM: 62)
          /usr/lib/gcc-cross/aarch64-linux-gnu/8/../../../../aarch64-linux-gnu/bin/ld: /opt/vector/target/aarch64-unknown-linux-gnu/release/deps/rustcBi5hre/librdkafka_sys-80e4439d10555fb9.rlib(rdkafka.o): Relocations in generic ELF (EM: 62)
          /usr/lib/gcc-cross/aarch64-linux-gnu/8/../../../../aarch64-linux-gnu/bin/ld: /opt/vector/target/aarch64-unknown-linux-gnu/release/deps/rustcBi5hre/librdkafka_sys-80e4439d10555fb9.rlib: error adding symbols: file in wrong format
          collect2: error: ld returned 1 exit status
          

error: could not compile `vector` (bin "vector") due to 1 previous error

.cargo/config.toml

[env]
# Build jemalloc assuming 64K pages so binaries also run on aarch64 hosts whose
# kernels ship with a 64K page size (notably EL8 aarch64). On 4K-page hosts this
# coarsens jemalloc's allocation granularity; no regression was measured in
# https://github.com/vectordotdev/vector/pull/18481.
JEMALLOC_SYS_WITH_LG_PAGE = "16"
CC_aarch64_unknown_linux_gnu = "aarch64-linux-gnu-gcc"
CXX_aarch64_unknown_linux_gnu = "aarch64-linux-gnu-g++"
RDKAFKA_CONFIGURE_FLAGS = "--host=aarch64-linux-gnu CC=aarch64-linux-gnu-gcc CXX=aarch64-linux-gnu-g++"
[target.aarch64-unknown-linux-gnu]
linker = "aarch64-linux-gnu-gcc"
rustflags = ["-C", "link-args=-rdynamic"]

[thomasqueirozb]

You can try to use cross for this. See https://github.com/vectordotdev/vector/blob/master/.github/workflows/cross.yml.

Note that we no longer use the aarch64 binary that is created by this workflow for our releases. We build on native arm instead. However, following what this workflow does locally should work

[FengZh61]

You can try to use cross for this. See https://github.com/vectordotdev/vector/blob/master/.github/workflows/cross.yml.

Note that we no longer use the aarch64 binary that is created by this workflow for our releases. We build on native arm instead. Following this workflow file should work regardless

thanks

[ChangkeunJ]

The decisive line in that wall of output is this one:

librdkafka_sys-...rlib(rdkafka.o): Relocations in generic ELF (EM: 62)

EM: 62 is EM_X86_64. So rdkafka.o inside that rlib was built for x86-64 while everything around it is aarch64, and the aarch64 ld won't touch it. Your cross toolchain is fine, librdkafka's own C build just never saw it.

RDKAFKA_CONFIGURE_FLAGS is where I'd look. Passing CC=aarch64-linux-gnu-gcc inside a flags string depends on rdkafka-sys splitting and forwarding it in a way that survives all the way to configure's compiler detection, and autotools generally wants --build alongside --host before it believes it's cross compiling at all. You can confirm it in about ten seconds rather than guessing: ar x the rlib and run file rdkafka.o. If it comes back x86-64 then you know you're fighting rdkafka-sys and not rustc.

There's a second one queued up right behind it, incidentally. -llua5.4 is sitting in your link line, and that resolves against your host's lua unless you've got an aarch64 lua5.4 to hand. Same wrong-format error, one crate later.

Which is the real argument for @thomasqueirozb's suggestion, and it happens to answer your glibc concern rather than sidestep it. Linking against libc6-dev-arm64-cross on a current distro produces a binary that wants that distro's glibc at runtime, so if the box you're deploying to is older it simply won't start. cross runs the build inside an image pinned to an old glibc, so the binary's minimum version comes out old as well. That's the property you're actually after, and it isn't something you can reach by adding flags to a local build.

[FengZh61]

root@debian:/opt/vector# find . -name rdkafka.o
./target/aarch64-unknown-linux-gnu/release/build/rdkafka-sys-0da974230f159343/out/src/rdkafka.o
root@debian:/opt/vector# file ./target/aarch64-unknown-linux-gnu/release/build/rdkafka-sys-0da974230f159343/out/src/rdkafka.o
./target/aarch64-unknown-linux-gnu/release/build/rdkafka-sys-0da974230f159343/out/src/rdkafka.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), with debug_info, not stripped

Yes, although aarch's target is used, rdkafka.o is still x86-64

[ChangkeunJ]

That settles it — and it also shows I sent you down a wrong turn. There is no RDKAFKA_CONFIGURE_FLAGS. I went back through build.rs and the only environment it reads is CARGO_FEATURE_*, the DEP_*_ROOT set, bare CFLAGS/LDFLAGS, OUT_DIR and a couple of cmake vars. Sorry for the detour.

What's actually going on is that rdkafka-sys has two entirely separate build bodies. Without the cmake-build feature it copies librdkafka into OUT_DIR and shells out to ./configure && make libs. That's librdkafka's own mklove, which reads bare CC and falls back to gcc — your host gcc. It never looks at CC_aarch64_unknown_linux_gnu. And mklove's --host/--build are registered purely as autoconf-compatibility no-ops, so passing those changes nothing either.

With cmake-build it goes through the cmake crate instead, which uses cc-rs, which does honour the CC_<target> family and sets CMAKE_SYSTEM_NAME/CMAKE_SYSTEM_PROCESSOR when the target differs from the host.

Vector turns that on for you, but only under its target features: target-base pulls in rdkafka?/cmake_build and target-aarch64-unknown-linux-gnu inherits it. default doesn't. So a plain cargo build --target aarch64-unknown-linux-gnu takes the mklove path and produces exactly the object you just found.

cargo clean -p rdkafka-sys --release --target aarch64-unknown-linux-gnu
export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc
export CC_aarch64_unknown_linux_gnu=aarch64-linux-gnu-gcc
export CXX_aarch64_unknown_linux_gnu=aarch64-linux-gnu-g++
export PKG_CONFIG_ALLOW_CROSS=1
cargo build --release --target aarch64-unknown-linux-gnu \
  --no-default-features --features target-aarch64-unknown-linux-gnu

The clean is not optional — the stale build directory gets reused and you'll get the same x86-64 object back. You'll want cmake, gcc-aarch64-linux-gnu and g++-aarch64-linux-gnu installed for it.

For context on why the earlier cross suggestion is still the safer road: the project doesn't cross-build this particular triple anymore, it builds aarch64 natively on ARM runners and keeps cross for the musl and armv7 targets. Both routes pass --no-default-features --features target-<triple>, which is where the cmake path comes from in the first place.

[FengZh61]

hi
It has been successfully compiled using the command line you provided and is running normally on the target machine. Thank you very much.