Irreproducible "index out of bounds" error in `rustc_thread_pool`

[zetanumbers]

This panic was triggered during crater run #146237 for parallel compiler enabled with 8 threads for crates azure_mgmt_datacatalog and bascule. The bascule crate has the default hello world code for binary crates (it was probably name squatting):

fn main() {
    println!("Hello, world!");
}

Upon executing a cargo command roughly like this:

cargo test --frozen --no-run --message-format json

The panic occured:

[INFO] [stderr] thread 'rustc' (30) panicked at compiler/rustc_thread_pool/src/sleep/mod.rs:354:52:
[INFO] [stderr] index out of bounds: the len is 8 but the index is 9223372036854775808

I couldn't personally reproduce this issue. The panic message and the unremarkable code make me think this is probably some very rare data race that could happen at anytime. As such I have no idea how would you fix this other than to very very thoroughly fuzz the rustc_thread_pool (aka rayon) crate.

I didn't check if previous crater runs ever had this error. So if there were any, that would be valuable to know. But I doubt that. Assuming that, it would give probability 2 out of 694485 crates.

Error log

https://crater-reports.s3.amazonaws.com/pr-146237/try%23d79d87d4c0f7f0d0785ad6e493cf8cf718ed89a6/reg/bascule-0.0.0/log.txt

[INFO] fetching crate bascule 0.0.0...
[INFO] testing bascule-0.0.0 against try#d79d87d4c0f7f0d0785ad6e493cf8cf718ed89a6 for pr-146237
[INFO] extracting crate bascule 0.0.0 into /workspace/builds/worker-1-tc2/source
[INFO] started tweaking crates.io crate bascule 0.0.0
[INFO] finished tweaking crates.io crate bascule 0.0.0
[INFO] tweaked toml for crates.io crate bascule 0.0.0 written to /workspace/builds/worker-1-tc2/source/Cargo.toml
[INFO] validating manifest of crates.io crate bascule 0.0.0 on toolchain d79d87d4c0f7f0d0785ad6e493cf8cf718ed89a6
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+d79d87d4c0f7f0d0785ad6e493cf8cf718ed89a6" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }`
[INFO] crate crates.io crate bascule 0.0.0 already has a lockfile, it will not be regenerated
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+d79d87d4c0f7f0d0785ad6e493cf8cf718ed89a6" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }`
[INFO] [stderr] warning: no edition set: defaulting to the 2015 edition while the latest is 2024
[INFO] [stderr]     Blocking waiting for file lock on package cache
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc2/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc2/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:7ad1b28ee6f5f7f699f6cf7015098d6ccdd96d6f2d78dd06228f5b4c9faf309c" "/opt/rustwide/cargo-home/bin/cargo" "+d79d87d4c0f7f0d0785ad6e493cf8cf718ed89a6" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }`
[INFO] [stdout] 28e25e9809fca9278e2f8279b56beee23acb34583562f910058a2e4b69421d5e
[INFO] running `Command { std: "docker" "start" "-a" "28e25e9809fca9278e2f8279b56beee23acb34583562f910058a2e4b69421d5e", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "28e25e9809fca9278e2f8279b56beee23acb34583562f910058a2e4b69421d5e", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "28e25e9809fca9278e2f8279b56beee23acb34583562f910058a2e4b69421d5e", kill_on_drop: false }`
[INFO] [stdout] 28e25e9809fca9278e2f8279b56beee23acb34583562f910058a2e4b69421d5e
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc2/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc2/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:7ad1b28ee6f5f7f699f6cf7015098d6ccdd96d6f2d78dd06228f5b4c9faf309c" "/opt/rustwide/cargo-home/bin/cargo" "+d79d87d4c0f7f0d0785ad6e493cf8cf718ed89a6" "build" "--frozen" "--message-format=json", kill_on_drop: false }`
[INFO] [stdout] cc5d7336b99327c001fc0f33f82270b5d5e8e58ffcad23dd676087677ee94031
[INFO] running `Command { std: "docker" "start" "-a" "cc5d7336b99327c001fc0f33f82270b5d5e8e58ffcad23dd676087677ee94031", kill_on_drop: false }`
[INFO] [stderr] warning: no edition set: defaulting to the 2015 edition while the latest is 2024
[INFO] [stderr]    Compiling bascule v0.0.0 (/opt/rustwide/workdir)
[INFO] [stderr]     Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.24s
[INFO] running `Command { std: "docker" "inspect" "cc5d7336b99327c001fc0f33f82270b5d5e8e58ffcad23dd676087677ee94031", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "cc5d7336b99327c001fc0f33f82270b5d5e8e58ffcad23dd676087677ee94031", kill_on_drop: false }`
[INFO] [stdout] cc5d7336b99327c001fc0f33f82270b5d5e8e58ffcad23dd676087677ee94031
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc2/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-1-tc2/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "1610612736" "--user" "0:0" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:7ad1b28ee6f5f7f699f6cf7015098d6ccdd96d6f2d78dd06228f5b4c9faf309c" "/opt/rustwide/cargo-home/bin/cargo" "+d79d87d4c0f7f0d0785ad6e493cf8cf718ed89a6" "test" "--frozen" "--no-run" "--message-format=json", kill_on_drop: false }`
[INFO] [stdout] fc76e1918fbc735e1644730e46b23ef22a0d167d0566eaedf07096528ee46cee
[INFO] running `Command { std: "docker" "start" "-a" "fc76e1918fbc735e1644730e46b23ef22a0d167d0566eaedf07096528ee46cee", kill_on_drop: false }`
[INFO] [stderr] warning: no edition set: defaulting to the 2015 edition while the latest is 2024
[INFO] [stderr]    Compiling bascule v0.0.0 (/opt/rustwide/workdir)
[INFO] [stderr] 
[INFO] [stderr] thread 'rustc' (30) panicked at compiler/rustc_thread_pool/src/sleep/mod.rs:354:52:
[INFO] [stderr] index out of bounds: the len is 8 but the index is 9223372036854775808
[INFO] [stderr] stack backtrace:
[INFO] [stderr]    0:     0x717c672785a2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h71e2dbb21440d8d7
[INFO] [stderr]    1:     0x717c67a0c8fc - core::fmt::write::ha34f2b5c70e30f63
[INFO] [stderr]    2:     0x717c6722ca41 - std::io::Write::write_fmt::h76264e5b71fd6d13
[INFO] [stderr]    3:     0x717c6723dfb2 - std::sys::backtrace::BacktraceLock::print::hb3aae3980c2c9bd5
[INFO] [stderr]    4:     0x717c672441c7 - std::panicking::default_hook::{{closure}}::h1b7baa88a148d351
[INFO] [stderr]    5:     0x717c67243cd3 - std::panicking::default_hook::hbc5e86c06039547d
[INFO] [stderr]    6:     0x717c662f78c1 - std[12c422ba0e8f2ccd]::panicking::update_hook::<alloc[9d95b5c0d0a090cb]::boxed::Box<rustc_driver_impl[2cee5112661b1264]::install_ice_hook::{closure#1}>>::{closure#0}
[INFO] [stderr]    7:     0x717c672446cf - std::panicking::panic_with_hook::ha6d2c09d64f936dd
[INFO] [stderr]    8:     0x717c6724447a - std::panicking::panic_handler::{{closure}}::h455c9d131d722636
[INFO] [stderr]    9:     0x717c6723e0f9 - std::sys::backtrace::__rust_end_short_backtrace::hd6ef955384ebd085
[INFO] [stderr]   10:     0x717c6721dd7d - __rustc[f338935add60802b]::rust_begin_unwind
[INFO] [stderr]   11:     0x717c64ae91f0 - core::panicking::panic_fmt::hb8d4a4959b79599f
[INFO] [stderr]   12:     0x717c658b1743 - core::panicking::panic_bounds_check::h4c590fbe220f041f
[INFO] [stderr]   13:     0x717c689ff2e2 - <rustc_thread_pool[5261d82459708a3a]::sleep::Sleep>::wake_specific_thread
[INFO] [stderr]   14:     0x717c68ce242b - <rustc_thread_pool[5261d82459708a3a]::job::HeapJob<<rustc_thread_pool[5261d82459708a3a]::scope::Scope>::spawn<rustc_interface[48cf4fa573a69c36]::passes::analysis::{closure#0}::{closure#2}::{closure#0}::{closure#2}::{closure#1}::{closure#0}::{closure#3}>::{closure#0}> as rustc_thread_pool[5261d82459708a3a]::job::Job>::execute
[INFO] [stderr]   15:     0x717c689ff82a - <rustc_thread_pool[5261d82459708a3a]::registry::WorkerThread>::wait_or_steal_until_cold
[INFO] [stderr]   16:     0x717c689fb6b5 - <rustc_thread_pool[5261d82459708a3a]::registry::ThreadBuilder>::run
[INFO] [stderr]   17:     0x717c689fb52d - std[12c422ba0e8f2ccd]::sys::backtrace::__rust_begin_short_backtrace::<<rustc_thread_pool[5261d82459708a3a]::ThreadPoolBuilder>::build_scoped<rustc_interface[48cf4fa573a69c36]::util::run_in_thread_pool_with_globals<rustc_interface[48cf4fa573a69c36]::interface::run_compiler<(), rustc_driver_impl[2cee5112661b1264]::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#5}::{closure#0}::{closure#0}, rustc_interface[48cf4fa573a69c36]::util::run_in_thread_pool_with_globals<rustc_interface[48cf4fa573a69c36]::interface::run_compiler<(), rustc_driver_impl[2cee5112661b1264]::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#5}::{closure#0}::{closure#1}, ()>::{closure#0}::{closure#0}::{closure#0}, ()>
[INFO] [stderr]   18:     0x717c689fa838 - <<std[12c422ba0e8f2ccd]::thread::Builder>::spawn_unchecked_<<rustc_thread_pool[5261d82459708a3a]::ThreadPoolBuilder>::build_scoped<rustc_interface[48cf4fa573a69c36]::util::run_in_thread_pool_with_globals<rustc_interface[48cf4fa573a69c36]::interface::run_compiler<(), rustc_driver_impl[2cee5112661b1264]::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#5}::{closure#0}::{closure#0}, rustc_interface[48cf4fa573a69c36]::util::run_in_thread_pool_with_globals<rustc_interface[48cf4fa573a69c36]::interface::run_compiler<(), rustc_driver_impl[2cee5112661b1264]::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#5}::{closure#0}::{closure#1}, ()>::{closure#0}::{closure#0}::{closure#0}, ()>::{closure#1} as core[18170c64b6a218e4]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
[INFO] [stderr]   19:     0x717c689f7435 - std::sys::pal::unix::thread::Thread::new::thread_start::h74991452e4c34e7b
[INFO] [stderr]   20:     0x717c6248aaa4 - <unknown>
[INFO] [stderr]   21:     0x717c62517a34 - clone
[INFO] [stderr]   22:                0x0 - <unknown>
[INFO] [stderr] 
[INFO] [stderr] error: the compiler unexpectedly panicked. this is a bug.
[INFO] [stderr] 
[INFO] [stderr] note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md
[INFO] [stderr] 
[INFO] [stderr] note: please make sure that you have updated to the latest nightly
[INFO] [stderr] 
[INFO] [stderr] warning: the ICE couldn't be written to `/opt/rustwide/workdir/rustc-ice-2025-09-11T03_13_47-25.txt`: Read-only file system (os error 30)
[INFO] [stderr] 
[INFO] [stderr] note: rustc 1.91.0-nightly (d79d87d4c 2025-09-05) running on x86_64-unknown-linux-gnu
[INFO] [stderr] 
[INFO] [stderr] note: compiler flags: -C embed-bitcode=no -C debuginfo=2
[INFO] [stderr] 
[INFO] [stderr] note: some of the compiler flags provided by cargo are hidden
[INFO] [stderr] 
[INFO] [stderr] query stack during panic:
[INFO] [stderr] end of query stack
[INFO] [stderr] Rayon: detected unexpected panic; aborting
[INFO] [stderr] error: could not compile `bascule` (bin "bascule" test)
[INFO] [stderr] 
[INFO] [stderr] Caused by:
[INFO] [stderr]   process didn't exit successfully: `/opt/rustwide/rustup-home/toolchains/d79d87d4c0f7f0d0785ad6e493cf8cf718ed89a6/bin/rustc --crate-name bascule --edition=2015 src/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --test --check-cfg 'cfg(docsrs,test)' --check-cfg 'cfg(feature, values())' -C metadata=33b4fa3f9b993a73 -C extra-filename=-1e496d20328f05df --out-dir /opt/rustwide/target/debug/deps -L dependency=/opt/rustwide/target/debug/deps --cap-lints=forbid` (signal: 6, SIGABRT: process abort signal)
[INFO] running `Command { std: "docker" "inspect" "fc76e1918fbc735e1644730e46b23ef22a0d167d0566eaedf07096528ee46cee", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "fc76e1918fbc735e1644730e46b23ef22a0d167d0566eaedf07096528ee46cee", kill_on_drop: false }`
[INFO] [stdout] fc76e1918fbc735e1644730e46b23ef22a0d167d0566eaedf07096528ee46cee

[zetanumbers]

Perhaps we could fallback to waking up every single thread just in case out of bounds happens? Although that would be just a patchwork, not the proper fix. Not to scare you but this bug might be more sinister than what we currently know.

[cuviper]

I believe this is a bug introduced in #142384, specifically here:

f52c6ee#diff-60dc6d86fa160643cb3c8cc8606ebc33c0211c4878b29076caa7267a74a430d3L393-R394

It turned worker_index into a reference, missing the subtlety of the note above that set may invalidate this.

rust/compiler/rustc_thread_pool/src/latch.rs

Lines 391 to 397 in 0be8e16

	// NOTE: Once we call `set` on the internal `latch`,
	// the target may proceed and invalidate `this`!
	match unsafe { &(*this).kind } {
	CountLatchKind::Stealing { latch, registry, worker_index } => {
	let registry = Arc::clone(registry);
	if unsafe { CoreLatch::set(latch) } {
	registry.notify_worker_latch_is_set(*worker_index);

So it's possible that the scope thread which owns that latch will be unblocked and kill it, and if that memory is written for anything else afterward, you could see a weird "index" like your 9223372036854775808 when you read it. This is definitely UB and would be very hard to reproduce in any case.

See #146799 for a fix, which I hope solves the entire issue here.

[cuviper]

As such I have no idea how would you fix this other than to very very thoroughly fuzz the rustc_thread_pool (aka rayon) crate.

BTW, please do CC me on anything for rustc_thread_pool that you suspect may apply to original rayon as well.