LTO coordinator panics by unwrapping Option::None

[knoellle]

Code

Discovered this error while fuzzing for #81280

I used cargo watch -- cargo build --profile incremental for building this binary crate, while touch-ing random files at random intervals to trigger rebuilds.

Fish script:

while true
    touch (find src | shuf -n1)
    sleep 0.(random 1 999)
end

Meta

rustc --version --verbose:

rustc 1.81.0 (eeb90cda1 2024-09-04)
binary: rustc
commit-hash: eeb90cda1969383f56a2637cbd3037bdf598841c
commit-date: 2024-09-04
host: x86_64-unknown-linux-gnu
release: 1.81.0
LLVM version: 18.1.7

Error output

thread 'coordinator' panicked at compiler/rustc_codegen_llvm/src/back/lto.rs:836:36:
called `Option::unwrap()` on a `None` value

Backtrace

called `Option::unwrap()` on a `None` value
stack backtrace:
   0:     0x7f5bbd68a3e5 - std::backtrace_rs::backtrace::libunwind::trace::h649ab3318d3445c5
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
   1:     0x7f5bbd68a3e5 - std::backtrace_rs::backtrace::trace_unsynchronized::hf4bb60c3387150c3
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7f5bbd68a3e5 - std::sys::backtrace::_print_fmt::hd9186c800e44bd00
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:65:5
   3:     0x7f5bbd68a3e5 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h1b9dad2a88e955ff
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:40:26
   4:     0x7f5bbd6d9eeb - core::fmt::rt::Argument::fmt::h351a7824f737a6a0
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/fmt/rt.rs:173:76
   5:     0x7f5bbd6d9eeb - core::fmt::write::h4b5a1270214bc4a7
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/fmt/mod.rs:1182:21
   6:     0x7f5bbd67ef6f - std::io::Write::write_fmt::hd04af345a50c312d
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/io/mod.rs:1827:15
   7:     0x7f5bbd68cbd1 - std::sys::backtrace::BacktraceLock::print::h68d41b51481bce5c
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:43:9
   8:     0x7f5bbd68cbd1 - std::panicking::default_hook::{{closure}}::h96ab15e9936be7ed
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:269:22
   9:     0x7f5bbd68c8ac - std::panicking::default_hook::h3cacb9c27561ad33
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:296:9
  10:     0x7f5bb9c93420 - std[1f2242ed6435445e]::panicking::update_hook::<alloc[7b1462a1eb55c293]::boxed::Box<rustc_driver_impl[8683aa37472b7dde]::install_ice_hook::{closure#0}>>::{closure#0}
  11:     0x7f5bbd68d59f - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hce7569f4ca5d1b64
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs:2084:9
  12:     0x7f5bbd68d59f - std::panicking::rust_panic_with_hook::hfe205f6954b2c97b
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:808:13
  13:     0x7f5bbd68d193 - std::panicking::begin_panic_handler::{{closure}}::h6cb44b3a50f28c44
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:667:13
  14:     0x7f5bbd68a8a9 - std::sys::backtrace::__rust_end_short_backtrace::hf1c1f2a92799bb0e
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:168:18
  15:     0x7f5bbd68ce54 - rust_begin_unwind
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:665:5
  16:     0x7f5bbd6d64a3 - core::panicking::panic_fmt::h3d8fc78294164da7
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panicking.rs:74:14
  17:     0x7f5bbd6d652c - core::panicking::panic::hec978767ec2d35ff
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panicking.rs:148:5
  18:     0x7f5bbd6d6299 - core::option::unwrap_failed::hba6b08832f9ce30b
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/option.rs:2020:5
  19:     0x7f5bbbf72a6b - rustc_codegen_llvm[5ff2375b9bf6d639]::back::lto::thin_lto
  20:     0x7f5bbbf6d00b - <rustc_codegen_llvm[5ff2375b9bf6d639]::LlvmCodegenBackend as rustc_codegen_ssa[382a85cbfbe8c2f6]::traits::write::WriteBackendMethods>::run_thin_lto
  21:     0x7f5bbc1113ce - rustc_codegen_ssa[382a85cbfbe8c2f6]::back::write::start_executing_work::<rustc_codegen_llvm[5ff2375b9bf6d639]::LlvmCodegenBackend>::{closure#5}
  22:     0x7f5bbc10f876 - std[1f2242ed6435445e]::sys::backtrace::__rust_begin_short_backtrace::<<rustc_codegen_llvm[5ff2375b9bf6d639]::LlvmCodegenBackend as rustc_codegen_ssa[382a85cbfbe8c2f6]::traits::backend::ExtraBackendMethods>::spawn_named_thread<rustc_codegen_ssa[382a85cbfbe8c2f6]::back::write::start_executing_work<rustc_codegen_llvm[5ff2375b9bf6d639]::LlvmCodegenBackend>::{closure#5}, core[3cad2706d8bdcdc4]::result::Result<rustc_codegen_ssa[382a85cbfbe8c2f6]::back::write::CompiledModules, ()>>::{closure#0}, core[3cad2706d8bdcdc4]::result::Result<rustc_codegen_ssa[382a85cbfbe8c2f6]::back::write::CompiledModules, ()>>
  23:     0x7f5bbc10f6ad - <<std[1f2242ed6435445e]::thread::Builder>::spawn_unchecked_<<rustc_codegen_llvm[5ff2375b9bf6d639]::LlvmCodegenBackend as rustc_codegen_ssa[382a85cbfbe8c2f6]::traits::backend::ExtraBackendMethods>::spawn_named_thread<rustc_codegen_ssa[382a85cbfbe8c2f6]::back::write::start_executing_work<rustc_codegen_llvm[5ff2375b9bf6d639]::LlvmCodegenBackend>::{closure#5}, core[3cad2706d8bdcdc4]::result::Result<rustc_codegen_ssa[382a85cbfbe8c2f6]::back::write::CompiledModules, ()>>::{closure#0}, core[3cad2706d8bdcdc4]::result::Result<rustc_codegen_ssa[382a85cbfbe8c2f6]::back::write::CompiledModules, ()>>::{closure#1} as core[3cad2706d8bdcdc4]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  24:     0x7f5bbd6975fb - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::ha1963004222e7822
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs:2070:9
  25:     0x7f5bbd6975fb - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h1086ced1f7c494c2
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs:2070:9
  26:     0x7f5bbd6975fb - std::sys::pal::unix::thread::Thread::new::thread_start::ha8af9c992ef0b208
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/pal/unix/thread.rs:108:17
  27:     0x7f5bbd466897 - start_thread
  28:     0x7f5bbd4eda5c - __GI___clone3
  29:                0x0 - <unknown>

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.81.0 (eeb90cda1 2024-09-04) running on x86_64-unknown-linux-gnu

note: compiler flags: --crate-type bin -C opt-level=3 -C embed-bitcode=no -C incremental=[REDACTED] -C strip=debuginfo

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
thread 'rustc' panicked at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/compiler/rustc_codegen_ssa/src/back/write.rs:2057:17:
/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/compiler/rustc_codegen_ssa/src/back/write.rs:2057:17: panic during codegen/LLVM phase
stack backtrace:
   0:     0x7f5bbd68a3e5 - std::backtrace_rs::backtrace::libunwind::trace::h649ab3318d3445c5
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
   1:     0x7f5bbd68a3e5 - std::backtrace_rs::backtrace::trace_unsynchronized::hf4bb60c3387150c3
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7f5bbd68a3e5 - std::sys::backtrace::_print_fmt::hd9186c800e44bd00
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:65:5
   3:     0x7f5bbd68a3e5 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h1b9dad2a88e955ff
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:40:26
   4:     0x7f5bbd6d9eeb - core::fmt::rt::Argument::fmt::h351a7824f737a6a0
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/fmt/rt.rs:173:76
   5:     0x7f5bbd6d9eeb - core::fmt::write::h4b5a1270214bc4a7
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/fmt/mod.rs:1182:21
   6:     0x7f5bbd67ef6f - std::io::Write::write_fmt::hd04af345a50c312d
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/io/mod.rs:1827:15
   7:     0x7f5bbd68cbd1 - std::sys::backtrace::BacktraceLock::print::h68d41b51481bce5c
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:43:9
   8:     0x7f5bbd68cbd1 - std::panicking::default_hook::{{closure}}::h96ab15e9936be7ed
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:269:22
   9:     0x7f5bbd68c8ac - std::panicking::default_hook::h3cacb9c27561ad33
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:296:9
  10:     0x7f5bb9c93420 - std[1f2242ed6435445e]::panicking::update_hook::<alloc[7b1462a1eb55c293]::boxed::Box<rustc_driver_impl[8683aa37472b7dde]::install_ice_hook::{closure#0}>>::{closure#0}
  11:     0x7f5bbd68d59f - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hce7569f4ca5d1b64
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs:2084:9
  12:     0x7f5bbd68d59f - std::panicking::rust_panic_with_hook::hfe205f6954b2c97b
                               at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:808:13
  13:     0x7f5bba236ae2 - std[1f2242ed6435445e]::panicking::begin_panic::<alloc[7b1462a1eb55c293]::string::String>::{closure#0}
  14:     0x7f5bba22f626 - std[1f2242ed6435445e]::sys::backtrace::__rust_end_short_backtrace::<std[1f2242ed6435445e]::panicking::begin_panic<alloc[7b1462a1eb55c293]::string::String>::{closure#0}, !>
  15:     0x7f5bba221201 - std[1f2242ed6435445e]::panicking::begin_panic::<alloc[7b1462a1eb55c293]::string::String>
  16:     0x7f5bba2b98cd - rustc_middle[ba2289ab3ae064d4]::util::bug::opt_span_bug_fmt::<rustc_span[28a649581f99a5bd]::span_encoding::Span>::{closure#0}
  17:     0x7f5bba29f49a - rustc_middle[ba2289ab3ae064d4]::ty::context::tls::with_opt::<rustc_middle[ba2289ab3ae064d4]::util::bug::opt_span_bug_fmt<rustc_span[28a649581f99a5bd]::span_encoding::Span>::{closure#0}, !>::{closure#0}
  18:     0x7f5bba29f31b - rustc_middle[ba2289ab3ae064d4]::ty::context::tls::with_context_opt::<rustc_middle[ba2289ab3ae064d4]::ty::context::tls::with_opt<rustc_middle[ba2289ab3ae064d4]::util::bug::opt_span_bug_fmt<rustc_span[28a649581f99a5bd]::span_encoding::Span>::{closure#0}, !>::{closure#0}, !>
  19:     0x7f5bb7c3a600 - rustc_middle[ba2289ab3ae064d4]::util::bug::bug_fmt
  20:     0x7f5bbc1081d0 - <rustc_codegen_llvm[5ff2375b9bf6d639]::LlvmCodegenBackend as rustc_codegen_ssa[382a85cbfbe8c2f6]::traits::backend::CodegenBackend>::join_codegen
  21:     0x7f5bbc1030c3 - <rustc_interface[53a414ae04dc6ffb]::queries::Linker>::link
  22:     0x7f5bbbf2d55c - rustc_interface[53a414ae04dc6ffb]::interface::run_compiler::<core[3cad2706d8bdcdc4]::result::Result<(), rustc_span[28a649581f99a5bd]::ErrorGuaranteed>, rustc_driver_impl[8683aa37472b7dde]::run_compiler::{closure#0}>::{closure#1}
  23:     0x7f5bbbe7395b - std[1f2242ed6435445e]::sys::backtrace::__rust_begin_short_backtrace::<rustc_interface[53a414ae04dc6ffb]::util::run_in_thread_with_globals<rustc_interface[53a414ae04dc6ffb]::interface::run_compiler<core[3cad2706d8bdcdc4]::result::Result<(), rustc_span[28a649581f99a5bd]::ErrorGuaranteed>, rustc_driver_impl[8683aa37472b7dde]::run_compiler::{closure#0}>::{closure#1}, core[3cad2706d8bdcdc4]::result::Result<(), rustc_span[28a649581f99a5bd]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[3cad2706d8bdcd

[saethlin]

while touch-ing random files at random intervals to trigger rebuilds.

FYI this will cause almost none of the incremental compilation code in the compiler to do anything interesting; random edits that are then reverted would tickle a lot more. It would also be slower.

It looks to me like you somehow ended up with an LLVM module (codegen unit) whose name is not UTF-8. That is wild. #130680 will improve the ICE message. If you're willing to apply that patch locally, I'd love to see what the module name you're coming up with is.

[knoellle]

Thanks for the hint regarding my methodology.

Im building the 1.81 release with your commit cherry-picked on top now according to these instruction. I have never built rust from source before, Am I correct in assuming that I can run x.py install with the prefix path set to an arbitrary directory which I then use in myl PATH before running my build command again?

Or will that detect that its not the same compiler anymore and try to build everything anew, losing the broken state?
If so, what would I need to do to pretend its still the same compiler version?

[saethlin]

I don't know anything about x install, I've never used it. I only know basically the quickstart steps: https://rustc-dev-guide.rust-lang.org/building/quickstart.html x build with rustup toolchain link lets you do cargo +stage1 build.

I'm also not sure I'd bother trying to make compiler not rebuild its artifacts. If you're fuzzing, eventually you should run into the same bad state again. Or you can show me how to run your fuzzer, and I'll run it overnight or on a cloud VM for a while.

[knoellle]

As suspected, the compiler didn't like the state and rebuilt everything. The reported error no longer occurs, but I'm trying to reproduce it now.

All of my fuzzing "tooling" and binary crate I was building (with commit hash) is in the issue description above if you want to run it yourself.
Basically just run the cargo watch in one shell and the while loop in another (fish) shell.

Since I'd guess its a timing issue, i.e. the compilation process being interrupted at some critical stage that isn't protected well enough, I'm not sure how well this will be reproducible on my machine (8845HS CPU) or even in a cloud VM. I'll also let this run over night, so we'll see.
I did make a backup of the tainted target directory that produces the issue with the 1.81 release though, just in case.

Edit: still nothing after eight hours. The first time it appeared after about three.

[knoellle]

Still nothing, been running this on two machines since my last post.

I did try to build a compiler that thinks its the same version as the 1.81.0 release and reuse the tainted artifacts.
I do have a compiler now that attempts to use the artifacts but I'm getting a wall of errors about basic identifiers not being found or crates being unknown. For this I was using the rustup toolchain link method.

Building from a clean state works with this compiler, it seems like it doesn't like mixing the standard libraries from different(ly named) toolchains, even though they should be identical. The commit hash the compiler version is tagged with is identical to the 1.81 release after I removed your cherry picked commit and just applied the change to the worktree.