Two concurrent cargo rustc calls hang

[sigiesec]

Problem
Quite often my firefox build gets stuck, I need to break it, restart, and by repeating this eventually does a complete build. I investigated one occurance of such a hang closer, and found there were two cargo processes running, started with cargo rustc --frozen --manifest-path.

The two processes are stuck in different states. The following are the backtraces of both processes:

(gdb) thread apply all bt

Thread 1 (Thread 0x7f90233d79c0 (LWP 26667)):
#0  0x00007f902360e60b in flock () at ../sysdeps/unix/syscall-template.S:78
#1  0x000055c19905aa2e in <std::fs::File as fs2::FileExt>::lock_exclusive ()
#2  0x000055c198ea3add in cargo::util::flock::Filesystem::open::{{closure}} ()
#3  0x000055c198ea415a in cargo::util::flock::acquire ()
#4  0x000055c198ea3934 in cargo::util::flock::Filesystem::open ()
#5  0x000055c198c72d99 in cargo::core::compiler::layout::Layout::at ()
#6  0x000055c198c72acb in cargo::core::compiler::layout::Layout::new ()
#7  0x000055c198f2a523 in cargo::core::compiler::context::Context::prepare_units ()
#8  0x000055c198f26855 in cargo::core::compiler::context::Context::compile ()
#9  0x000055c198fe275f in cargo::ops::cargo_compile::compile_ws ()
#10 0x000055c198fe1077 in cargo::ops::cargo_compile::compile ()
#11 0x000055c198c0311c in cargo::commands::rustc::exec ()
#12 0x000055c198be507a in cargo::cli::main ()
#13 0x000055c198c0cb1b in cargo::main ()
#14 0x000055c198c24673 in std::rt::lang_start::{{closure}} ()
#15 0x000055c199489dc3 in std::rt::lang_start_internal::{{closure}} () at src/libstd/rt.rs:49
#16 std::panicking::try::do_call () at src/libstd/panicking.rs:293
#17 0x000055c199495aaa in __rust_maybe_catch_panic () at src/libpanic_unwind/lib.rs:85
#18 0x000055c19948a98d in std::panicking::try () at src/libstd/panicking.rs:272
#19 std::panic::catch_unwind () at src/libstd/panic.rs:394
#20 std::rt::lang_start_internal () at src/libstd/rt.rs:48
#21 0x000055c198c0f1a2 in main ()

(gdb) thread apply all bt

Thread 1 (Thread 0x7fde387b29c0 (LWP 26666)):
#0  0x00007fde389ed588 in __GI___poll (fds=0x7ffc20bc5d28, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000055a13e76cee5 in jobserver::imp::Client::acquire ()
#2  0x000055a13e7706a9 in jobserver::Client::acquire_raw ()
#3  0x000055a13e1bf187 in cargo::util::flock::acquire ()
#4  0x000055a13e1be934 in cargo::util::flock::Filesystem::open ()
#5  0x000055a13df8dd99 in cargo::core::compiler::layout::Layout::at ()
#6  0x000055a13df8dacb in cargo::core::compiler::layout::Layout::new ()
#7  0x000055a13e245523 in cargo::core::compiler::context::Context::prepare_units ()
#8  0x000055a13e241855 in cargo::core::compiler::context::Context::compile ()
#9  0x000055a13e2fd75f in cargo::ops::cargo_compile::compile_ws ()
#10 0x000055a13e2fc077 in cargo::ops::cargo_compile::compile ()
#11 0x000055a13df1e11c in cargo::commands::rustc::exec ()
#12 0x000055a13df0007a in cargo::cli::main ()
#13 0x000055a13df27b1b in cargo::main ()
#14 0x000055a13df3f673 in std::rt::lang_start::{{closure}} ()
#15 0x000055a13e7a4dc3 in std::rt::lang_start_internal::{{closure}} () at src/libstd/rt.rs:49
#16 std::panicking::try::do_call () at src/libstd/panicking.rs:293
#17 0x000055a13e7b0aaa in __rust_maybe_catch_panic () at src/libpanic_unwind/lib.rs:85
#18 0x000055a13e7a598d in std::panicking::try () at src/libstd/panicking.rs:272
#19 std::panic::catch_unwind () at src/libstd/panic.rs:394
#20 std::rt::lang_start_internal () at src/libstd/rt.rs:48
#21 0x000055a13df2a1a2 in main ()

Steps

1. run ./mach build on mozilla-unified/central

Possible Solution(s)
No idea.

Notes

Output of cargo version:

[osboxes@sigiesec-mozbln mozilla-unified]$ /home/osboxes/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo version
cargo 1.36.0 (c4fcfb725 2019-05-15)

This is running on Fedora 30.

[alexcrichton]

Thanks for the report and stack traces! This is definitely a bug in Cargo which is basically a deadlock introduced in #6748, so I'll post a PR to revert that soon.