Building rust 1.75.0 fails on NetBSD when using an external LLVM due to missing RPATH

[he32]

Code

I am trying to keep up with rust releases, cross-building bootstrap kits for the various NetBSD targets using cross-compilation from a NetBSD/amd64 host. While doing this, I have been using the rust-supplied amd64 binary bootstrap kit to bootstrap the amd64 rust compiler. That has worked up until 1.74.1 (using the 1.73.0 bootstrap kit), but now fails using the 1.74.1 bootstrap kit to build 1.75.0.

I expected to see this happen: that the build would succeed.

Instead, this happened: The build progresses until "stage1" as far as I can see. Here is the part of the build log which surrounds the build error:

     Running `/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/bootstrap/debug/rustc --crate-name rustc_main --edition=2021 compiler/rustc/src/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="llvm"' --cfg 'feature="max_level_info"' -Zunstable-options --check-cfg 'values(feature, "jemalloc", "jemalloc-sys", "llvm", "max_level_info", "rustc_use_parallel_compiler")' --check-cfg 'names()' --check-cfg 'values()' -C metadata=99ab614842099465 -C extra-filename=-99ab614842099465 --out-dir /usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage0-rustc/x86_64-unknown-netbsd/release/deps --target x86_64-unknown-netbsd -L dependency=/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage0-rustc/x86_64-unknown-netbsd/release/deps -L dependency=/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage0-rustc/release/deps --extern rustc_codegen_ssa=/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage0-rustc/x86_64-unknown-netbsd/release/deps/librustc_codegen_ssa-f10772593a84d8ca.rlib --extern rustc_driver=/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage0-rustc/x86_64-unknown-netbsd/release/deps/librustc_driver-d78e0abcd3c75aa5.so --extern rustc_driver_impl=/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage0-rustc/x86_64-unknown-netbsd/release/deps/librustc_driver_impl-df6994cba8710313.rlib --extern rustc_smir=/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage0-rustc/x86_64-unknown-netbsd/release/deps/librustc_smir-7fee53f16ca15c1a.rlib --extern stable_mir=/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage0-rustc/x86_64-unknown-netbsd/release/deps/libstable_mir-9381a3ae56e93b67.rlib --cfg=bootstrap --cfg=windows_raw_dylib -Csymbol-mangling-version=v0 -Zunstable-options '--check-cfg=values(bootstrap)' '--check-cfg=values(parallel_compiler)' '--check-cfg=values(no_btreemap_remove_entry)' '--check-cfg=values(crossbeam_loom)' '--check-cfg=values(span_locations)' '--check-cfg=values(rustix_use_libc)' '--check-cfg=values(emulate_second_only_system)' '--check-cfg=values(windows_raw_dylib)' -Zmacro-backtrace -Clink-args=-Wl,-z,origin '-Clink-args=-Wl,-rpath,$ORIGIN/../lib' -Zunstable-options -Csplit-debuginfo=off -Zunstable-options '-Wrustc::internal' -Cprefer-dynamic -Z binary-dep-depinfo -L native=/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage0-rustc/x86_64-unknown-netbsd/release/build/psm-2407b8a67950d553/out -L native=/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage0-rustc/x86_64-unknown-netbsd/release/build/rustc_llvm-13a2ebc4724c6674/out -L native=/usr/pkg/lib -L native=/usr/lib -L native=/usr/pkg/lib`
    Finished release [optimized] target(s) in 4m 52s
Creating a sysroot for stage1 compiler (use `rustup toolchain link 'name' build/host/stage1`)
Removing sysroot /usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage1 to avoid caching bugs
Removing sysroot /usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage1/lib/rustlib/x86_64-unknown-netbsd/lib to avoid caching bugs
Dirty - /usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage1-std
Building stage1 library artifacts (x86_64-unknown-netbsd)
running: cd "/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src" && env -u MAKEFLAGS -u MFLAGS AR_x86_64_unknown_netbsd="ar" CARGO_INCREMENTAL="0" CARGO_PROFILE_RELEASE_DEBUG="0" CARGO_PROFILE_RELEASE_DEBUG_ASSERTIONS="false" CARGO_PROFILE_RELEASE_OVERFLOW_CHECKS="false" CARGO_TARGET_DIR="/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage1-std" CC_x86_64_unknown_netbsd="gcc" CFG_COMPILER_HOST_TRIPLE="x86_64-unknown-netbsd" CFG_DISABLE_UNSTABLE_FEATURES="1" CFG_RELEASE_CHANNEL="stable" CFLAGS_x86_64_unknown_netbsd="-O2 -ffunction-sections -fdata-sections -fPIC -m64 -O2 -I/usr/pkg/include -I/usr/include -I/usr/pkg/include/python3.11" CXXFLAGS_x86_64_unknown_netbsd="-O2 -ffunction-sections -fdata-sections -fPIC -m64 -O2 -I/usr/pkg/include -I/usr/include -I/usr/pkg/include/python3.11" CXX_x86_64_unknown_netbsd="c++" LIBC_CHECK_CFG="1" RANLIB_x86_64_unknown_netbsd="ar s" REAL_LIBRARY_PATH="/usr/pkgsrc/wip/rust175/work/rust-bootstrap/lib" REAL_LIBRARY_PATH_VAR="LD_LIBRARY_PATH" RUSTBUILD_NATIVE_DIR="/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/native" RUSTC="/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/bootstrap/debug/rustc" RUSTC_BOOTSTRAP="1" RUSTC_BREAK_ON_ICE="1" RUSTC_ERROR_METADATA_DST="/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/tmp/extended-error-metadata" RUSTC_FORCE_UNSTABLE="1" RUSTC_HOST_FLAGS="-Zunstable-options --check-cfg=values(bootstrap)" RUSTC_INSTALL_BINDIR="bin" RUSTC_LIBDIR="/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage1/lib" RUSTC_LINT_FLAGS="-Wrust_2018_idioms -Wunused_lifetimes -Wsemicolon_in_expressions_from_macros -Dwarnings" RUSTC_REAL="/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage1/bin/rustc" RUSTC_SNAPSHOT="/usr/pkgsrc/wip/rust175/work/rust-bootstrap/bin/rustc" RUSTC_SNAPSHOT_LIBDIR="/usr/pkgsrc/wip/rust175/work/rust-bootstrap/lib" RUSTC_STAGE="1" RUSTC_SYSROOT="/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage1" RUSTC_VERBOSE="1" RUSTDOC="/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/bootstrap/debug/rustdoc" RUSTDOCFLAGS="-Csymbol-mangling-version=legacy -Zunstable-options -Zunstable-options --check-cfg=values(bootstrap) --check-cfg=values(stdarch_intel_sde) --check-cfg=values(no_fp_fmt_parse) --check-cfg=values(no_global_oom_handling) --check-cfg=values(no_rc) --check-cfg=values(no_sync) --check-cfg=values(freebsd12) --check-cfg=values(freebsd13) --check-cfg=values(backtrace_in_libstd) --check-cfg=values(target_env,\"libnx\") --check-cfg=values(target_arch,\"asmjs\",\"spirv\",\"nvptx\",\"xtensa\") -Dwarnings -Wrustdoc::invalid_codeblock_attributes --crate-version 1.75.0\t(82e1608df\t2023-12-21)\t(built\tfrom\ta\tsource\ttarball) -Zcrate-attr=doc(html_root_url=\"https://doc.rust-lang.org/1.75.0/\") -Zcrate-attr=warn(rust_2018_idioms)" RUSTDOC_REAL="/path/to/nowhere/rustdoc/not/required" RUSTFLAGS="-Csymbol-mangling-version=legacy -Zunstable-options -Zunstable-options --check-cfg=values(bootstrap) --check-cfg=values(stdarch_intel_sde) --check-cfg=values(no_fp_fmt_parse) --check-cfg=values(no_global_oom_handling) --check-cfg=values(no_rc) --check-cfg=values(no_sync) --check-cfg=values(freebsd12) --check-cfg=values(freebsd13) --check-cfg=values(backtrace_in_libstd) --check-cfg=values(target_env,\"libnx\") --check-cfg=values(target_arch,\"asmjs\",\"spirv\",\"nvptx\",\"xtensa\") -Zmacro-backtrace -Clink-args=-Wl,-z,origin -Clink-args=-Wl,-rpath,$ORIGIN/../lib -Zunstable-options -Csplit-debuginfo=off -Cprefer-dynamic -Zinline-mir -Cembed-bitcode=yes -Zcrate-attr=doc(html_root_url=\"https://doc.rust-lang.org/1.75.0/\")" RUST_COMPILER_RT_ROOT="/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/src/llvm-project/compiler-rt" RUST_TEST_THREADS="32" WINAPI_NO_BUNDLED_LIBRARIES="1" __CARGO_DEFAULT_LIB_METADATA="stablestd" "/usr/pkgsrc/wip/rust175/work/rust-bootstrap/bin/cargo" "build" "--target" "x86_64-unknown-netbsd" "--release" "-Zcheck-cfg=names,values,output" "-Zbinary-dep-depinfo" "-j" "32" "-v" "--frozen" "--features" " panic-unwind backtrace compiler-builtins-c" "--manifest-path" "/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/library/sysroot/Cargo.toml" "--message-format" "json-render-diagnostics"
error: process didn't exit successfully: `/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/bootstrap/debug/rustc -vV` (exit status: 1)
--- stdout

Did not run successfully: exit status: 1
LD_LIBRARY_PATH="/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage1/lib:/usr/pkgsrc/wip/rust175/work/rust-bootstrap/lib" "/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage1/bin/rustc" "-vV" "-Wrust_2018_idioms" "-Wunused_lifetimes" "-Wsemicolon_in_expressions_from_macros" "-Dwarnings" "-Zunstable-options" "--check-cfg=values(bootstrap)" "-Z" "force-unstable-if-unmarked"
-------------

--- stderr
/usr/pkgsrc/wip/rust175/work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage1/lib/librustc_driver-d78e0abcd3c75aa5.so: Shared object "libLLVM-16.so" not found

Building my own "bootstrap kit" for 1.74.1 using our pkgsrc setup produces a bootstrap kit which succeeds in building rust 1.75.0.

Looking at the build log above it is evident that the rust-supplied 1.74.1 bootstrap kit is built against an external LLVM, not the internal LLVM embedded in the rust distribution (ref. the version; the embedded LLVM is 17.x), and possibly that the rust bootstrap kit is not built with the --enable-cargo-native-static configure option(?) which according to my understanding minimizes or at least reduces the external dynamic dependencies. It is possible that these two conspire in contribution to this error.

Version it worked on

The bootstrap kit for x86_64-unknown-netbsd version 1.73.0 worked to build 1.74.1.

Version with regression

Trying to use the 1.74.1 bootstrap kit for x86_64-unknown-netbsd to build 1.75.0 fails, as above.

The following is for the working version that I built:

rustc --version --verbose:

: {1} setenv LD_LIBRARY_PATH /usr/pkgsrc/wip/rust175/work/rust-bootstrap/lib/
...
: {4} work/rust-bootstrap/bin/rustc --version --verbose
rustc 1.74.1 (a28077b28 2023-12-04) (built from a source tarball)
binary: rustc
commit-hash: a28077b28a02b92985b3a3faecf92813155f1ea1
commit-date: 2023-12-04
host: x86_64-unknown-netbsd
release: 1.74.1
LLVM version: 17.0.4
: {5}

This one does not have a dynamic dependency on LLVM:

: {5} ldd work/rust-bootstrap/bin/rustc
work/rust-bootstrap/bin/rustc:
        -lrustc_driver-23f6dd35cae7ab09 => /usr/pkgsrc/wip/rust175/work/rust-bootstrap/lib//librustc_driver-23f6dd35cae7ab09.so
        -lstd-00afc197bbba9b83 => /usr/pkgsrc/wip/rust175/work/rust-bootstrap/lib//libstd-00afc197bbba9b83.so
        -lpthread.1 => /usr/lib/libpthread.so.1
        -lc.12 => /usr/lib/libc.so.12
        -lgcc_s.1 => /usr/lib/libgcc_s.so.1
        -lexecinfo.0 => /usr/lib/libexecinfo.so.0
        -lelf.2 => /usr/lib/libelf.so.2
        -lm.0 => /usr/lib/libm.so.0
        -lz.1 => /usr/lib/libz.so.1
        -lstdc++.9 => /usr/lib/libstdc++.so.9
: {6} 

and

: {7} readelf -a /usr/pkgsrc/wip/rust175/work/rust-bootstrap/lib//librustc_driver-23f6dd35cae7ab09.so | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libstd-00afc197bbba9b83.so]
 0x0000000000000001 (NEEDED)             Shared library: [libexecinfo.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.9]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.12]
: {8} 

(this is the version I've built using our pkgsrc setup and which can be used to build 1.75.0.)

[Mark-Simulacrum]

What do you mean by "bootstrap kit"? The rustc tarball for 1.74.1 links against the LLVM within that tarball:

$ curl -O https://static.rust-lang.org/dist/rustc-1.74.1-x86_64-unknown-linux-gnu.tar.xz
$ tar xf rustc-1.74.1-x86_64-unknown-linux-gnu.tar.xz
$ cd rustc-1.74.1-x86_64-unknown-linux-gnu/rustc/bin/
$ ldd rustc
        linux-vdso.so.1 (0x00007fffdd9f2000)
        librustc_driver-9ad8050f1171bcfc.so => /home/mark/rustc-1.74.1-x86_64-unknown-linux-gnu/rustc/bin/./../lib/librustc_driver-9ad8050f1171bcfc.so (0x00007f461cfbe000)
        libstd-b149a04e58514815.so => /home/mark/rustc-1.74.1-x86_64-unknown-linux-gnu/rustc/bin/./../lib/libstd-b149a04e58514815.so (0x00007f461ce26000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f461ce12000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f461ce0d000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f461ce08000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f461cbdd000)
        libLLVM-17-rust-1.74.1-stable.so => /home/mark/rustc-1.74.1-x86_64-unknown-linux-gnu/rustc/bin/./../lib/../lib/libLLVM-17-rust-1.74.1-stable.so (0x00007f4615200000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f461cbbd000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f4615119000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f46213e5000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f461cba1000)

[he32]

Sorry, "bootstrap kit" is perhaps pkgsrc vocabulary, it's the "previous rust version" which is needed to build the version in question.

For the NetBSD amd64 bits I basically see the same as you, i.e. no dynamic dependency on an external LLVM.

However, the error is real, though I see that the diagnosis is perhaps faulty. The error occurs as soon as the "stage1" compiler is being used. I'll revert my workaround locally and produce a bit more diagnostics.

[he32]

It appears that I will have to "crawl to the cross" and admit that I no longer appear able to reproduce this problem: when reverting back to using rust's binary distribution for x86_64-unknown-netbsd the build now succeeds. Sorry for the noise, I'll just close this issue.

[he32]

OK, so I've looked into this a bit closer. The title is a bit misleading, and it doesn't look like the "bootstrap" compiler (the previous version) is the problem.

Instead, it looks like the problem occurs when building the rust compiler with an external LLVM. If I look at the output of "readelf -a" for 1.74.1 built with an external LLVM, I get:

% readelf -a /usr/pkg/lib/librustc_driver-d49d1997f5743dd5.so
...
Dynamic section at offset 0x41d80b0 contains 31 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libstd-3cce52205d7c2a20.so]
 0x0000000000000001 (NEEDED)             Shared library: [libLLVM-16.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.9]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.12]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.0]
 0x000000000000000f (RPATH)              Library rpath: [/usr/pkg/lib]
 0x000000000000000c (INIT)               0x583770
...

However, if I do the same for the stage1 librustc_driver during the 1.75.0 rust build, I get:

% readelf -a work/rustc-1.75.0-src/build/x86_64-unknown-netbsd/stage1/lib/librustc_driver-fe36ec5db90badd9.so
...
Dynamic section at offset 0x4340f40 contains 30 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libstd-26c8033e83cc1c07.so]
 0x0000000000000001 (NEEDED)             Shared library: [libLLVM-16.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.9]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.12]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.0]
 0x000000000000000c (INIT)               0x5ab200
...

and there is no trace of any RPATH in the shared object at all. So of course ldd and attempts at executing the stage1 rustc is met with "I cannot find libLLVM-16.so", and on this platform you must supply RPATH to locate any "external" libraries.

This is also not an issue with $ORIGIN being suppressed in pkgsrc, since the external LLVM does not exist in the stage1 lib directory, but "elsewhere" when you build with an external LLVM.

I'll see if I can change the title of this issue to be more reflective of the actual problem.