lld doesn't work on NixOS

[DaniD3v]

Code

Compilation for the cargo hello-world template with the new nightly failed.

cargo build
   Compiling abc v0.1.0 (/tmp/abc)
error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/self-contained:/home/notyou/.cargo/bin:/home/notyou/.local/bin:/run/wrappers/bin:/home/notyou/.nix-profile/bin:/nix/profile/bin:/home/notyou/.local/state/nix/profile/bin:/etc/profiles/per-user/notyou/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin:.local/bin:/nix/store/m3gq2f6iwh3g67zrs27m7w82b37l8gf8-binutils-wrapper-2.41/bin:/nix/store/a4qsjxipd9jgjdy89fggzq9nlkf2m8jq-pciutils-3.12.0/bin:/nix/store/md6hh4rrcrf99nssvcam3qaqs3skj086-gcc-wrapper-13.2.0/bin" VSLANG="1033" "cc" "-m64" "/tmp/rustcZqYSP3/symbols.o" "/tmp/abc/target/debug/deps/abc-76f7a6c126880f8e.15uilqnmz9fl0pdsf1pp09837.rcgu.o" "/tmp/abc/target/debug/deps/abc-76f7a6c126880f8e.2dblsqlv26pnrb2gnxxzoafd0.rcgu.o" "/tmp/abc/target/debug/deps/abc-76f7a6c126880f8e.6ocuuw51mfoea9wtaypp2vi8j.rcgu.o" "/tmp/abc/target/debug/deps/abc-76f7a6c126880f8e.72b8ipzr9dmhbx6yeplx0a04m.rcgu.o" "/tmp/abc/target/debug/deps/abc-76f7a6c126880f8e.9ewb0tvybbd504u6turswtxvi.rcgu.o" "/tmp/abc/target/debug/deps/abc-76f7a6c126880f8e.dn1mnlgzn93y0wymda22rajod.rcgu.o" "/tmp/abc/target/debug/deps/abc-76f7a6c126880f8e.3ct60f9m81ggqnk1jmipk7pg8.rcgu.o" "-Wl,--as-needed" "-L" "/tmp/abc/target/debug/deps" "-L" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-de48b8168d6cf4fa.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-9f3477fb95a0bba7.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-210d920812faea91.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-f3d3451767410a17.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-1a79dd36d08251de.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-886230e7120831b2.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-2caea079085a58a2.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-a9d7d97cdc65a449.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-5727477b0a78105a.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-7e555563aa211118.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-c8f13465f1a795b2.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-ee5b5774583426df.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-14df174c91007922.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-e8bfe52be756260a.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-6275035a459b3ada.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-b6892f3c52c68f01.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-f72b956e24d1de70.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-632ae0f28c5e55ff.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-e8b7e96e438f08f6.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-B/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld" "-fuse-ld=lld" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-L" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/self-contained" "-o" "/tmp/abc/target/debug/deps/abc-76f7a6c126880f8e" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
  = note: collect2: fatal error: execvp: No such file or directory
          compilation terminated.


error: could not compile `abc` (bin "abc") due to 1 previous error

It compiles when I either switch to the stable toolchain,
or set the RUSTFLAGS="-Z linker-features=-lld" env variable

Version it worked on

It most recently worked on: nightly-2024-05-17

Version with regression

rustc --version --verbose:

rustc --version --verbose
rustc 1.80.0-nightly (d84b90375 2024-05-19)
binary: rustc
commit-hash: d84b9037541f45dc2c52a41d723265af211c0497
commit-date: 2024-05-19
host: x86_64-unknown-linux-gnu
release: 1.80.0-nightly
LLVM version: 18.1.4

Backtrace

Backtrace

   Compiling abc v0.1.0 (/tmp/abc)
error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/self-contained:/home/notyou/.cargo/bin:/home/notyou/.local/bin:/run/wrappers/bin:/home/notyou/.nix-profile/bin:/nix/profile/bin:/home/notyou/.local/state/nix/profile/bin:/etc/profiles/per-user/notyou/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin:.local/bin:/nix/store/m3gq2f6iwh3g67zrs27m7w82b37l8gf8-binutils-wrapper-2.41/bin:/nix/store/a4qsjxipd9jgjdy89fggzq9nlkf2m8jq-pciutils-3.12.0/bin:/nix/store/md6hh4rrcrf99nssvcam3qaqs3skj086-gcc-wrapper-13.2.0/bin" VSLANG="1033" "cc" "-m64" "/tmp/rustcJFepHM/symbols.o" "/tmp/abc/target/debug/deps/abc-76f7a6c126880f8e.15uilqnmz9fl0pdsf1pp09837.rcgu.o" "/tmp/abc/target/debug/deps/abc-76f7a6c126880f8e.2dblsqlv26pnrb2gnxxzoafd0.rcgu.o" "/tmp/abc/target/debug/deps/abc-76f7a6c126880f8e.6ocuuw51mfoea9wtaypp2vi8j.rcgu.o" "/tmp/abc/target/debug/deps/abc-76f7a6c126880f8e.72b8ipzr9dmhbx6yeplx0a04m.rcgu.o" "/tmp/abc/target/debug/deps/abc-76f7a6c126880f8e.9ewb0tvybbd504u6turswtxvi.rcgu.o" "/tmp/abc/target/debug/deps/abc-76f7a6c126880f8e.dn1mnlgzn93y0wymda22rajod.rcgu.o" "/tmp/abc/target/debug/deps/abc-76f7a6c126880f8e.3ct60f9m81ggqnk1jmipk7pg8.rcgu.o" "-Wl,--as-needed" "-L" "/tmp/abc/target/debug/deps" "-L" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-de48b8168d6cf4fa.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-9f3477fb95a0bba7.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-210d920812faea91.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-f3d3451767410a17.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-1a79dd36d08251de.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-886230e7120831b2.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-2caea079085a58a2.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-a9d7d97cdc65a449.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-5727477b0a78105a.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-7e555563aa211118.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-c8f13465f1a795b2.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-ee5b5774583426df.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-14df174c91007922.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-e8bfe52be756260a.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-6275035a459b3ada.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-b6892f3c52c68f01.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-f72b956e24d1de70.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-632ae0f28c5e55ff.rlib" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-e8b7e96e438f08f6.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-B/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld" "-fuse-ld=lld" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-L" "/home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/self-contained" "-o" "/tmp/abc/target/debug/deps/abc-76f7a6c126880f8e" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
  = note: collect2: fatal error: execvp: No such file or directory
          compilation terminated.
          

error: could not compile `abc` (bin "abc") due to 1 previous error

[DaniD3v]

related: (same error)
#71515

[pnkfelix]

cc @lqd (at least, I know they were driving the "use lld by default" effort a la rust-lang/compiler-team#510 )

presumably this was injected by #124129

[lqd]

I don't know NixOS, so I'm not going to be of much help. I believe it has issues with lld in general, for example NixOS/nixpkgs#24744 that could be related, or this could be a duplicate of that issue. Changes such as NixOS/nixpkgs#24744 (comment) could be needed to make both lld and rust-lld work there.

cc may not be able to find lld because the toolchain is incomplete (the "No such file or directory" part of the error) or because nix's configuration prevents it (the execvp part of the error, I guess).

To see if we can rule out the first case, I'd like to know the contents of the directories we tell cc to look for the lld wrappers calling rust-lld:

• /home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin: it should at least contain the rust-lld binary, and the following gcc-ld subfolder.
• and /home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld: it should contain the 3/4 target-specific lld executables.

If the contents of the directories match what we expect, I would guess the issue is more NixOS/nixpkgs#24744.

[DaniD3v]

• /home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin: it should at least contain the rust-lld binary, and the following gcc-ld subfolder.

It does:

ls /home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin

gcc-ld  rust-lld

• and /home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld: it should contain the 3/4 target-specific lld executables.

Don't know if that requirement is satisfied:

ls /home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld

ld64.lld  ld.lld  lld-link  wasm-ld

If this issue is nix-specific we should probably make it extra clear what the work-around is until the issue is resolved.

I added this to my ~/.cargo/config.toml:

[target.x86_64-unknown-linux-gnu]
rustflags = ["-Zlinker-features=-lld"] # fix nightly bug

[lqd]

The expected executables are there, the toolchain looks complete AFAICT. Again I don't know NixOs, but there are wasm and aarch64 targets that already use rust-lld by default, so I would expect these to fail there, regardless of #124129.

Similarly, we now just do what users did to change the linker, with regular link args like -Clink-arg=-fuse-ld=lld, as well as adding a -B path like /home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld. One can emulate what rustc does, and that needs to work, does it?

The way to revert to the default linker like you did is indeed explicitly mentioned in the blog post announcing the change.

It seems less like a nightly bug to me than a known nix-specific issue.

[Nemo157]

How are you installing the nightly? It erroring out at that stage seems like it's not "fixing" the ld.lld binary's link paths. Using oxalica/rust-overlay it manages to correctly execute ld.lld, but then if the binary it builds uses any native dependencies it fails because the linker isn't "fixing" the binaries it builds (which is NixOS/nixpkgs#24744 related, oxalica/rust-overlay#168).

[Nemo157]

If it's via nixpkgs-rustup, I assume this patch needs updating as it doesn't match these new file paths: https://github.com/NixOS/nixpkgs/blob/4cc02342ff90b8484f11e68bdac985837bb956f9/pkgs/development/tools/rust/rustup/0001-dynamically-patchelf-binaries.patch (but that will just get it to the same state as oxalica/rust-overlay, the self-contained lld will execute but not fully work correctly).

[Nemo157]

Nixpkgs issue: NixOS/nixpkgs#312661

[DaniD3v]

How are you installing the nightly?

Using rustup.