bootstrap is broken for all custom targets on rust-1.85.0_beta

[stefson]

hello everyone,

I've been using a custom target for my locally built rustc toolchain for many years now, the target is added with a custom patch and everything has been working just fine:

rustc 1.84.0 (9fc6b4312 2025-01-07) (built from a source tarball)
binary: rustc
commit-hash: 9fc6b43126469e3858e2fe86cafb4f0fd5068869
commit-date: 2025-01-07
host: armv7a-unknown-linux-musleabihf
release: 1.84.0
LLVM version: 19.1.5

rustc --print target-list | grep armv7a-unknown
> armv7a-unknown-linux-musleabihf

with the branching of the 1.85.0_beta branch, the already built-in target of the local rustc-1.84.0 binary cannot be found anymore by the bootstrap process:

running: /var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/build/bootstrap/debug/bootstrap build -vv --config=/var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/config.toml -j1
WARNING: The `change-id` is missing in the `config.toml`. This means that you will not be able to track the major changes made to the bootstrap configurations.
NOTE: to silence this warning, add `change-id = 134650` at the top of `config.toml`
finding compilers


error occurred: unknown target `armv7a-unknown-linux-musleabihf`


Traceback (most recent call last):
  File "/var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/./x.py", line 53, in <module>
    bootstrap.main()
  File "/var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/src/bootstrap/bootstrap.py", line 1333, in main
    bootstrap(args)
  File "/var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/src/bootstrap/bootstrap.py", line 1308, in bootstrap
    run(args, env=env, verbose=build.verbose, is_bootstrap=True)
  File "/var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/src/bootstrap/bootstrap.py", line 236, in run
    raise RuntimeError(err)
RuntimeError: failed to run: /var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/build/bootstrap/debug/bootstrap build -vv --config=/var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/config.toml -j1

the full build.log: build.log.gz

this is my config.toml:

[llvm]
download-ci-llvm = false
optimize = true
release-debuginfo = false
assertions = false
ninja = true
targets = "ARM"
experimental-targets = ""
link-shared = false
[build]
build = "armv7a-unknown-linux-musleabihf"
host = ["armv7a-unknown-linux-musleabihf"]
target = ["armv7a-unknown-linux-musleabihf"]
cargo = "/usr/lib/rust/1.84.0/bin/cargo"
rustc = "/usr/lib/rust/1.84.0/bin/rustc"
docs = false
compiler-docs = false
submodules = false
python = "python3.12"
locked-deps = true
vendor = true
extended = true
tools = ["cargo","rustdoc"]
verbose = 2
sanitizers = false
cargo-native-static = false
[install]
prefix = "/usr/lib/rust/1.85.0_beta"
sysconfdir = "etc"
docdir = "share/doc/rust"
bindir = "bin"
libdir = "lib"
mandir = "share/man"
[rust]
# https://github.com/rust-lang/rust/issues/54872
codegen-units-std = 1
optimize = true
debug = false
debug-assertions = false
debuginfo-level-rustc = 0
backtrace = true
incremental = false
default-linker = "armv7a-unknown-linux-musleabihf-gcc"
channel = "stable"
rpath = false
verbose-tests = true
optimize-tests = true
codegen-tests = true
dist-src = false
# https://github.com/rust-lang/rust/issues/117885
remap-debuginfo = true
lld = false
backtrace-on-ice = true
jemalloc = false
# See https://github.com/rust-lang/rust/issues/121124
lto = "off"
[dist]
src-tarball = false
compression-formats = ["xz"]
[target.armv7a-unknown-linux-musleabihf]
cc = "armv7a-unknown-linux-musleabihf-gcc"
cxx = "armv7a-unknown-linux-musleabihf-g++"
linker = "armv7a-unknown-linux-musleabihf-gcc"
ar = "armv7a-unknown-linux-musleabihf-ar"
crt-static = false

operating system is gentoo, both armv7a and amd64

here are the backtraces:

RUST_BACKTRACE=1 /var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/build/bootstrap/debug/bootstrap build -vv --config=/var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/config.toml -j1
thread 'main' panicked at src/bootstrap/src/bin/main.rs:38:43:
fs::OpenOptions::new().write(true).truncate(true).create(true).open(&lock_path) failed with No such file or directory (os error 2)
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: bootstrap::main
             at ./work/rustc-beta-src/src/bootstrap/src/bin/main.rs:38:43
   3: core::ops::function::FnOnce::call_once
             at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

 RUST_BACKTRACE=full /var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/build/bootstrap/debug/bootstrap build -vv --config=/var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/config.toml -j1
thread 'main' panicked at src/bootstrap/src/bin/main.rs:38:43:
fs::OpenOptions::new().write(true).truncate(true).create(true).open(&lock_path) failed with No such file or directory (os error 2)
stack backtrace:
   0:   0xf306d0 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hc18560b2598d9e23
   1:   0xf5e460 - core::fmt::write::h1d71005846ea8a6f
   2:   0xf2b7f0 - std::io::Write::write_fmt::h0ee65933c12a137e
   3:   0xf30564 - std::sys::backtrace::BacktraceLock::print::h12961e334efd2170
   4:   0xf31ee4 - std::panicking::default_hook::{{closure}}::h5a0b1665890f1cc2
   5:   0xf31cdc - std::panicking::default_hook::h314f25ba09977fe2
   6:   0xf32634 - std::panicking::rust_panic_with_hook::hbfa151c65a0e5210
   7:   0xf32324 - std::panicking::begin_panic_handler::{{closure}}::h97bdd921809c6f05
   8:   0xf30c40 - std::sys::backtrace::__rust_end_short_backtrace::hf1a1e2e30f5a34d8
   9:   0xf3204c - rust_begin_unwind
  10:   0x45d1a4 - core::panicking::panic_fmt::hf8aa1f743ac38530
  11:   0x462e80 - bootstrap::main::h7df149a85fdac2da
                       at /var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/src/bootstrap/src/bin/main.rs:38:43
  12:   0x462084 - core::ops::function::FnOnce::call_once::he4f8c341d28a4588
                       at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ops/function.rs:250:5
  13:   0x466904 - std::sys::backtrace::__rust_begin_short_backtrace::h8b2447e5dd3de0c0
                       at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/sys/backtrace.rs:154:18
  14:   0x461e18 - std::rt::lang_start::{{closure}}::hfef738705190f40a
                       at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/rt.rs:195:18
  15:   0xf2137c - std::rt::lang_start_internal::h5a47ea8ee334112b
  16:   0x461df0 - std::rt::lang_start::h501fb101ec03ed78
                       at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/rt.rs:194:17
  17:   0x464670 - main

[onur-ozkan]

Can you check if setting BOOTSTRAP_SKIP_TARGET_SANITY=1 make it work?

[stefson]

like this?

 BOOTSTRAP_SKIP_TARGET_SANITY=1 /var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/build/bootstrap/debug/bootstrap build -vv --config=/var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/config.toml -j1

[onur-ozkan]

You should use bootstrap scripts (x or x.py) instead of bare bootstrap binary.

[stefson]

ok, so here is the result of

BOOTSTRAP_SKIP_TARGET_SANITY=1 RUST_BACKTRACE=full RUSTC_BOOTSTRAP=1 python3.12 ./x.py build -vv --config=./config.toml -j1

    Finished `dev` profile [unoptimized] target(s) in 13m 04s
running: /var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/build/bootstrap/debug/bootstrap build -vv --config=/var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/config.toml -j1
WARNING: The `change-id` is missing in the `config.toml`. This means that you will not be able to track the major changes made to the bootstrap configurations.
NOTE: to silence this warning, add `change-id = 134650` at the top of `config.toml`
finding compilers


error occurred: unknown target `armv7a-unknown-linux-musleabihf`


Traceback (most recent call last):
  File "/var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/./x.py", line 53, in <module>
    bootstrap.main()
  File "/var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/src/bootstrap/bootstrap.py", line 1338, in main
    bootstrap(args)
  File "/var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/src/bootstrap/bootstrap.py", line 1313, in bootstrap
    run(args, env=env, verbose=build.verbose, is_bootstrap=True)
  File "/var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/src/bootstrap/bootstrap.py", line 236, in run
    raise RuntimeError(err)
RuntimeError: failed to run: /var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/build/bootstrap/debug/bootstrap build -vv --config=/var/tmp/portage/dev-lang/rust-1.85.0_beta/work/rustc-beta-src/config.toml -j1

so that doesn't make any difference, I'm afraid.

[arachsys]

I'm seeing this too on the now released rust-1.85.0, with (longstanding) {aarch64,x86_64}-$VENDOR-linux-musl targets added in compiler/rustc_target/src/spec/targets/ and in the table in compiler/rustc_target/src/spec/mod.rs. The custom target worked fine up to and including rust-1.84.1, but aborts on bootstrap with rust-1.85.0.

(To preempt worries that targets/*-$VENDOR-linux-musl.rs are invalid, I can reproduce when they are literally a copy of the corresponding *-unknown-linux-musl.rs. Nowadays my vendor target's only difference is to remove the incorrect static-link default on my native musl system, so there's nothing tricky going on.)

A bit of println!() debugging suggests we fail on

let compiler = cfg.get_compiler();

of find_target() in src/bootstrap/src/utils/cc_detect.rs.

Explicitly specifying cc = "/bin/cc" etc. in the target section of config.toml doesn't change this behaviour, and strace suggests bootstrap execs rustc --version --verbose to get the target name, but never tries to run cc, c++, ar, etc.

[arachsys]

This seems to be related to the cc-rs mistake discussed in rust-lang/cc-rs#1317 - this introduced a hard-coded list of targets (!!) which broke custom rust packaging on Alpine Linux, Chimera Linux and others, though it hadn't made it all the way out to break a 'stable' rust release at the point they all reported it.

The bootstrap build works fine if you bodge the hard-coded list in the vendored cc crate:

diff --git a/vendor/cc-1.2.0/src/target/generated.rs b/vendor/cc-1.2.0/src/target/generated.rs
index 0c9b0ae1..23d18039 100644
--- a/vendor/cc-1.2.0/src/target/generated.rs
+++ b/vendor/cc-1.2.0/src/target/generated.rs
@@ -124,6 +124,18 @@ pub(crate) const LIST: &[(&str, TargetInfo<'static>)] = &[
             unversioned_llvm_target: "arm64-apple-watchos-simulator",
         },
     ),
+    (
+        "aarch64-arachsys-linux-musl",
+        TargetInfo {
+            full_arch: "aarch64",
+            arch: "aarch64",
+            vendor: "arachsys",
+            os: "linux",
+            env: "musl",
+            abi: "",
+            unversioned_llvm_target: "aarch64-arachsys-linux-musl",
+        },
+    ),
     (
         "aarch64-fuchsia",
         TargetInfo {
@@ -2848,6 +2860,18 @@ pub(crate) const LIST: &[(&str, TargetInfo<'static>)] = &[
             unversioned_llvm_target: "x86_64-apple-watchos-simulator",
         },
     ),
+    (
+        "x86_64-arachsys-linux-musl",
+        TargetInfo {
+            full_arch: "x86_64",
+            arch: "x86_64",
+            vendor: "arachsys",
+            os: "linux",
+            env: "musl",
+            abi: "",
+            unversioned_llvm_target: "x86_64-arachsys-linux-musl",
+        },
+    ),
     (
         "x86_64-fortanix-unknown-sgx",
         TargetInfo {

[stefson]

I tried to copy your approach:

diff --git a/vendor/cc-1.2.0/src/target/generated.rs b/vendor/cc-1.2.0/src/target/generated.rs
index 0c9b0ae..ee14246 100644
--- a/vendor/cc-1.2.0/src/target/generated.rs
+++ b/vendor/cc-1.2.0/src/target/generated.rs
@@ -832,6 +832,18 @@ pub(crate) const LIST: &[(&str, TargetInfo<'static>)] = &[
             unversioned_llvm_target: "armv7-unknown-linux-gnueabi",
         },
     ),
+    (
+        "armv7a-unknown-linux-musleabihf",
+        TargetInfo {
+            full_arch: "armv7a",
+            arch: "arm",
+            vendor: "unknown",
+            os: "linux",
+            env: "musl",
+            abi: "eabihf",
+            unversioned_llvm_target: "armv7-unknown-linux-gnueabihf",
+        },
+    ),
     (
         "armv7-unknown-linux-musleabihf",
         TargetInfo {

sadly it didn't work for me, and failed with the same error as before.

[arachsys]

Ah, sorry to hear that - possibly something about the way I build rust allows this to work for me and not for you? My build recipe is at https://github.com/arachsys/packages/tree/master/rust and you can see the diff I needed to get the 1.85.0 release tarball building (with both 1.84.1 and itself) at arachsys/packages@7596f3fac1. (The only relevant bit of that diff for you is probably the change to target.diff, and I can't see anything that might make me too 'unusual' in the build script itself.)

One thing I didn't think of last night: you'll definitely have to work around the ridiculous file checksums if you patch vendor/cc-* in the source tree with upstream cargo. You can just remove the entire "files" key from them with sed:

sed -i -E 'H;1h;$!d;x;s/("files"\s*:\s*\{)("([^"\\]|\\.)*"|[^"}])*/\1/' vendor/*/.cargo-checksum.json

or disable this (deliberately user-hostile) behaviour altogether with something like arachsys/packages@bdc6635b5e.

[heitbaum]

How are we progressing with a fix for this build issue?

[onur-ozkan]

Can you downgrade bootstrap cc to =1.1.22 and check if it works? I hope other versions of cc from compiler and library tree will not effect this.