Trouble setting `default-linker = ...` when building aarch64-unknown-linux-musl Rust target

[kpcyrd]

Hello!

I'm building the 1.79.0 Rust compiler for an x86_64 host together with the aarch64-unknown-linux-musl target and I'm trying to pass the following configuration:

[target.aarch64-unknown-linux-musl]
sanitizers = false
cc = "/usr/bin/aarch64-linux-gnu-gcc"
default-linker = "/usr/aarch64-linux-musl/bin/musl-gcc"
musl-root = "/usr/aarch64-linux-musl/lib/musl"

This mostly works, but when trying to build a binary it's attempting to use a cc binary to link the aarch64 binary (which fails):

$ cargo build --target aarch64-unknown-linux-musl
   Compiling hello v0.1.0 (/work/hello)
error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/usr/lib64/rustlib/x86_64-unknown-linux-gnu/bin:/usr/lib64/rustlib/x86_64-unknown-linux-gnu/bin/self-contained:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" VSLANG="1033" "cc" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crt1.o" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crti.o" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtbegin.o" "/tmp/rustc2CHM1q/symbols.o" "/work/hello/target/aarch64-unknown-linux-musl/debug/deps/hello-7931b1584cd1ab67.1xmk4uifdlwcyvcf.rcgu.o" "/work/hello/target/aarch64-unknown-linux-musl/debug/deps/hello-7931b1584cd1ab67.2iot206naxak0i8c.rcgu.o" "/work/hello/target/aarch64-unknown-linux-musl/debug/deps/hello-7931b1584cd1ab67.36ysn34pesl14418.rcgu.o" "/work/hello/target/aarch64-unknown-linux-musl/debug/deps/hello-7931b1584cd1ab67.3sofp44czwr2ps5f.rcgu.o" "/work/hello/target/aarch64-unknown-linux-musl/debug/deps/hello-7931b1584cd1ab67.48fdjopboqsydtmb.rcgu.o" "/work/hello/target/aarch64-unknown-linux-musl/debug/deps/hello-7931b1584cd1ab67.53mzv8vau8okjd0w.rcgu.o" "/work/hello/target/aarch64-unknown-linux-musl/debug/deps/hello-7931b1584cd1ab67.399dl2mpfvoomi3h.rcgu.o" "-Wl,--as-needed" "-L" "/work/hello/target/aarch64-unknown-linux-musl/debug/deps" "-L" "/work/hello/target/debug/deps" "-L" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/libstd-3596dae639322199.rlib" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/libpanic_unwind-42f36a2be85eb6dc.rlib" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/libobject-92b4428c10de0dc9.rlib" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/libmemchr-69adc492ebb97bd4.rlib" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/libaddr2line-ca6a40a49fe3c882.rlib" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/libgimli-3302655791f1d018.rlib" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/librustc_demangle-da279c3e643e3217.rlib" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/libstd_detect-8da93d3a6c5174b1.rlib" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/libhashbrown-ea8be399915822e4.rlib" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/librustc_std_workspace_alloc-eeda7694da25b729.rlib" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/libminiz_oxide-c44e1c5c2140de11.rlib" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/libadler-81560391c9d2bf23.rlib" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/libunwind-dbebdcfd8e2ea004.rlib" "-lunwind" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/libcfg_if-58f9a769bf33f884.rlib" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/liblibc-0ad51dfa48c68161.rlib" "-lc" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/liballoc-6aaf9613f369bde4.rlib" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/librustc_std_workspace_core-c52a370e4bdaf372.rlib" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/libcore-fa10a65c5a02736f.rlib" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/libcompiler_builtins-6b454192e1e41575.rlib" "-Wl,-Bdynamic" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-nostartfiles" "-L" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib" "-L" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/self-contained" "-o" "/work/hello/target/aarch64-unknown-linux-musl/debug/deps/hello-7931b1584cd1ab67" "-Wl,--gc-sections" "-static" "-no-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtend.o" "/usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtn.o"
  = note: /usr/sbin/ld: /usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crt1.o: Relocations in generic ELF (EM: 183)
          /usr/sbin/ld: /usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crt1.o: Relocations in generic ELF (EM: 183)
          /usr/sbin/ld: /usr/lib64/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crt1.o: error adding symbols: file in wrong format
          collect2: error: ld returned 1 exit status
          

error: could not compile `hello` (bin "hello") due to 1 previous error

From what I understand this shouldn't happen because I'm explicitly providing a different default. The config.toml also has a setting linker = ... but according to the docs this value is only used for bootstrapping, while default-linker = ... is the default value that gets hard-coded into the compiler binary (which is what I want).

I tried strings /usr/bin/rustc and couldn't find any configuration unfortunately.

When manually passing -C linker=... through RUSTFLAGS the hello-world binary builds successfully:

$ RUSTFLAGS="-Clinker=/usr/aarch64-linux-musl/bin/musl-gcc" cargo build --target aarch64-unknown-linux-musl
   Compiling hello v0.1.0 (/work/hello)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.24s

Alternatively this works too, but I'd prefer having it work out of the box:

$ cat > ~/.cargo/config.toml
[target.aarch64-unknown-linux-musl]
linker = "/usr/aarch64-linux-musl/bin/musl-gcc"
$ cargo build --target aarch64-unknown-linux-musl --release
   Compiling hello v0.1.0 (/work/hello)
    Finished `release` profile [optimized] target(s) in 0.26s

When copying the binary over to my aarch64 test system it runs correctly:

$ ./hello.bin 
Hello, world!

So the musl part seems to be setup correctly and the only missing piece is how I make Rust pick the right linker by default.

Full config.toml

# see src/bootstrap/defaults/
profile = "dist"

# see src/bootstrap/src/utils/change_tracker.rs
change-id = 123711

[llvm]
link-shared = true

[build]
target = [
  "x86_64-unknown-linux-gnu",
  "i686-unknown-linux-gnu",
  "x86_64-unknown-linux-musl",
  "aarch64-unknown-linux-musl",
  "wasm32-unknown-unknown",
  "wasm32-wasi",
  "wasm32-wasip1",
  "wasm32-wasip1-threads",
  "wasm32-wasip2",
]
cargo = "/usr/bin/cargo"
rustc = "/usr/bin/rustc"
rustfmt = "/usr/bin/rustfmt"
locked-deps = true
vendor = true
tools = [
  "cargo",
  "clippy",
  "rustdoc",
  "rustfmt",
  "rust-analyzer-proc-macro-srv",
  "analysis",
  "src",
  "rust-demangler",
]
sanitizers = true
profiler = true

# Generating docs fails with the wasm32-* targets
docs = false

[install]
prefix = "/usr"

[rust]
codegen-units-std = 1
debuginfo-level = 1
debuginfo-level-std = 2
channel = "stable"
description = "Arch Linux rust 1:1.79.0-4"
rpath = false
frame-pointers = true
deny-warnings = false
backtrace-on-ice = true
remap-debuginfo = false
jemalloc = true

[dist]
compression-formats = ["gz"]

[target.x86_64-unknown-linux-gnu]
cc = "/usr/bin/gcc"
cxx = "/usr/bin/g++"
ar = "/usr/bin/gcc-ar"
ranlib = "/usr/bin/gcc-ranlib"
llvm-config = "/usr/bin/llvm-config"

[target.i686-unknown-linux-gnu]
cc = "/usr/bin/gcc"
cxx = "/usr/bin/g++"
ar = "/usr/bin/gcc-ar"
ranlib = "/usr/bin/gcc-ranlib"

[target.x86_64-unknown-linux-musl]
sanitizers = false
musl-root = "/usr/lib/musl"

[target.aarch64-unknown-linux-musl]
sanitizers = false
cc = "/usr/bin/aarch64-linux-gnu-gcc"
default-linker = "/usr/aarch64-linux-musl/bin/musl-gcc"
musl-root = "/usr/aarch64-linux-musl/lib/musl"

[target.wasm32-unknown-unknown]
sanitizers = false
profiler = false

[target.wasm32-wasi]
sanitizers = false
profiler = false
wasi-root = "/usr/share/wasi-sysroot"

[target.wasm32-wasip1]
sanitizers = false
profiler = false
wasi-root = "/usr/share/wasi-sysroot"

[target.wasm32-wasip1-threads]
sanitizers = false
profiler = false
wasi-root = "/usr/share/wasi-sysroot"

[target.wasm32-wasip2]
sanitizers = false
profiler = false
wasi-root = "/usr/share/wasi-sysroot"

[GrigorenkoPV]

Shouldn't the cc also be linux-musl and not linux-gnu?

[kpcyrd]

It doesn't really seem to matter in this case, but I've updated my config to this:

[target.aarch64-unknown-linux-musl]
sanitizers = false
cc = "/usr/aarch64-linux-musl/bin/musl-gcc"
default-linker = "/usr/aarch64-linux-musl/bin/musl-gcc"
musl-root = "/usr/aarch64-linux-musl/lib/musl"

It builds but the issue persists, I'm keeping this anyway since this seems to be "the more correct" approach, thanks!

[bjorn3]

What does RUSTC_BOOTSTRAP=1 rustc --target aarch64-unknown-linux-musl -Zunstable-options --print target-spec-json show for the rustc you built?

[kpcyrd]

{
  "arch": "aarch64",
  "crt-objects-fallback": "musl",
  "crt-static-default": true,
  "crt-static-respected": true,
  "data-layout": "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128",
  "dynamic-linking": true,
  "env": "musl",
  "features": "+v8a",
  "has-rpath": true,
  "has-thread-local": true,
  "is-builtin": true,
  "linker-flavor": "gnu-cc",
  "llvm-target": "aarch64-unknown-linux-musl",
  "max-atomic-width": 128,
  "metadata": {
    "description": null,
    "host_tools": null,
    "std": null,
    "tier": null
  },
  "os": "linux",
  "position-independent-executables": true,
  "post-link-objects-fallback": {
    "dynamic-dylib": [
      "crtendS.o",
      "crtn.o"
    ],
    "dynamic-nopic-exe": [
      "crtend.o",
      "crtn.o"
    ],
    "dynamic-pic-exe": [
      "crtendS.o",
      "crtn.o"
    ],
    "static-dylib": [
      "crtendS.o",
      "crtn.o"
    ],
    "static-nopic-exe": [
      "crtend.o",
      "crtn.o"
    ],
    "static-pic-exe": [
      "crtendS.o",
      "crtn.o"
    ]
  },
  "pre-link-objects-fallback": {
    "dynamic-dylib": [
      "crti.o",
      "crtbeginS.o"
    ],
    "dynamic-nopic-exe": [
      "crt1.o",
      "crti.o",
      "crtbegin.o"
    ],
    "dynamic-pic-exe": [
      "Scrt1.o",
      "crti.o",
      "crtbeginS.o"
    ],
    "static-dylib": [
      "crti.o",
      "crtbeginS.o"
    ],
    "static-nopic-exe": [
      "crt1.o",
      "crti.o",
      "crtbegin.o"
    ],
    "static-pic-exe": [
      "rcrt1.o",
      "crti.o",
      "crtbeginS.o"
    ]
  },
  "relro-level": "full",
  "stack-probes": {
    "kind": "inline"
  },
  "supported-sanitizers": [
    "address",
    "leak",
    "memory",
    "thread",
    "cfi"
  ],
  "supported-split-debuginfo": [
    "packed",
    "unpacked",
    "off"
  ],
  "supports-xray": true,
  "target-family": [
    "unix"
  ],
  "target-mcount": "\u0001_mcount",
  "target-pointer-width": "64"
}

[bjorn3]

Seems like CFG_DEFAULT_LINKER wasn't set at all while compiling rustc for whatever reason.

[kpcyrd]

There's now a downstream patch in Arch Linux to make this work: https://gitlab.archlinux.org/archlinux/packaging/packages/rust/-/blob/11737403916ce22c6c53557190ba44fae13a2ca5/0005-compiler-Use-aarch64-linux-gnu-gcc-to-link-aarch64-t.patch

[heftig]

I'm not sure how the default-linker-per-target that's advertised is supposed to work since the rustc_target crate is part of the compiler and only compiled once, thus there's only one CFG_DEFAULT_LINKER for all spec files.