cargo panics with "timeout in poll?" error from jobserver

[durka]

Here's a log trying to build clippy: https://gist.github.com/b721066be81601774e16087210ad1919

The error happens every time at a random point while building dependencies. The only way to fix it is to disable parallelism with -j 1.

This might be a bug in jobserver, or OSX itself. I'm on OSX 10.9.5.

[alexcrichton]

I can't find anything on google that points to 10.9 specifically (everyone seems to claim 10.12 is broken) and all the "brokenness" appears to be in different ways? If we want to fix this we probably just want to have a fallback to select in cases like this.

[durka]

Yeah, all I found was the 10.12 stuff as well. I couldn't get a simple repro by just calling libc::poll. Do you have any idea what kind of scenario cargo is creating (calling poll a bunch of times from separate threads, or something)?

…

On Thu, Oct 19, 2017 at 10:37 AM, Alex Crichton ***@***.***> wrote: I can't find anything on google that points to 10.9 specifically (everyone seems to claim 10.12 is broken) and all the "brokenness" appears to be in different ways? If we want to fix this we probably just want to have a fallback to select in cases like this. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4643 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC3n1lXZsV_la_BREwPmkrKOubvvkDvks5st16pgaJpZM4P-peZ> .

[alexcrichton]

AFAIK the only "funky" thing here is that the fds are shared across multiple processes, but other than that there's not a whole lot going on here?

[ivanceras]

+1 on OSX Yosemite 10.10.3

cargo -V
cargo 0.23.0-nightly (e447ac7e9 2017-09-27)
rustc -V
rustc 1.22.0-nightly (4750c1ec0 2017-10-19)

panic error:

thread '<unnamed>' panicked at 'timeout in poll?', /Users/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/jobserver-0.1.7/src/lib.rs:505:29

[alexcrichton]

I'm going to hope that rust-lang/jobserver-rs@400ce8c fixes this.

[durka]

Let me know if/how I can help test.

…

On Fri, Oct 20, 2017 at 1:29 PM, Alex Crichton ***@***.***> wrote: I'm going to hope that ***@***.*** <rust-lang/jobserver-rs@400ce8c> fixes this. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4643 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC3nzAyu9P-8grDVOnq0UeO5NcLIMgpks5suNhrgaJpZM4P-peZ> .

[alexcrichton]

Can you compile Cargo from source? If so, can you try compiling Cargo with that patch and see if it fixes things?

[durka]

OK, I'll let you know how that goes. Can I make rustup use a custom cargo and my installed rustc or how does that work?

…

On Fri, Oct 20, 2017 at 1:35 PM, Alex Crichton ***@***.***> wrote: Can you compile Cargo from source? If so, can you try compiling Cargo with that patch and see if it fixes things? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4643 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC3n002cxpU2iCU3fQ_WSc1DHFjspPMks5suNn9gaJpZM4P-peZ> .

[alexcrichton]

You'd probably just do something like rustup run beta path/to/target/release/cargo build and that should work ok

[durka]

When I do that it's somehow still running rustup's version of cargo... not sure what's going on.

…

On Fri, Oct 20, 2017 at 1:47 PM, Alex Crichton ***@***.***> wrote: You'd probably just do something like rustup run beta path/to/target/release/cargo build and that should work ok — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4643 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC3n-RthJi8XBUudaaLE-QuhbzgLAWXks5suNy2gaJpZM4P-peZ> .

[durka]

Oh no the problem is that librustc_trans also depends on jobserver :( so I need to build the whole compiler as well.

…

On Fri, Oct 20, 2017 at 1:59 PM, Alex Burka ***@***.***> wrote: When I do that it's somehow still running rustup's version of cargo... not sure what's going on. On Fri, Oct 20, 2017 at 1:47 PM, Alex Crichton ***@***.***> wrote: > You'd probably just do something like rustup run beta > path/to/target/release/cargo build and that should work ok > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#4643 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAC3n-RthJi8XBUudaaLE-QuhbzgLAWXks5suNy2gaJpZM4P-peZ> > . >

[durka]

@alexcrichton maybe this issue is in the wrong repo now, but I can't get rustc to build with that patch.

I'm running ./x.py build -i --stage 1 src/libtest. It fails while compiling stage1 test artifacts with "timeout in poll?" Fine. I can then run ./x.py build -i --stage 1 src/libtest -j 1 to complete the build.

I edited src/Cargo.toml to override jobserver with the git repo. Now it still fails during stage1 test artifacts, but attempting to continue with -j 1 yields

Building stage1 std artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
   Compiling std_unicode v0.0.0 (file:///Users/alex/Programming/rust/rust/src/libstd_unicode)
thread '<unnamed>' panicked at 'failed to shut down worker thread', /Users/alex/.cargo/git/checkouts/jobserver-rs-27359438239da508/400ce8c/src/lib.rs:641:16
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::begin_panic
   6: jobserver::imp::Helper::join
   7: <jobserver::HelperThread as core::ops::drop::Drop>::drop
   8: core::ptr::drop_in_place
   9: core::ptr::drop_in_place
error: Error during translation/LLVM phase.

error: Could not compile `std_unicode`.

[alexcrichton]

Ah oops I think I botched the patch. Forgot that EINTR needs to come out as an error and not get handled. Mind testing rust-lang/jobserver-rs@b3e6165?

[durka]

After rebuilding rustc and cargo with that patch it seems to be working!

…

On Fri, Oct 20, 2017 at 3:19 PM, Alex Crichton ***@***.***> wrote: Ah oops I think I botched the patch. Forgot that EINTR needs to come out as an error and not get handled. Mind testing alexcrichton/jobserver-rs@ b3e6165 <rust-lang/jobserver-rs@b3e6165> ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4643 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC3n8lxmBqozx5Hl6hZzbb2wNlMllFWks5suPJEgaJpZM4P-peZ> .

[alexcrichton]

Ok thanks for checking! I've pushed a new version of jobserver so all that's left is to send a PR to rust-lang/rust updating the crate