Timeout occurs during build

[hellow554]

It seems to me, that the backend is not fast enough to actually compile the code anymore.

It seems to be random, therefore maybe load dependent, but I sometimes get a timeout even with a simple fn main() {}.

Can you investigate that? I'm not sure how to reproduce it, because as said, it is somewhat random and does not depend on the code itself.

[shepmaster]

Yep. I think it's been this way for a few months now. I haven't had time to figure it out, or even to add some basic metrics to help in figuring it out. It may just be that Rust / Cargo / the various crates are slower.

It's a 2x Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz and ~8GiB of RAM.

[josephlr]

I've also been hitting this recently, but running with miri usually doesn't hit this. Is running w/ Miri less CPU intensive?

EDIT: Seems like running w/ miri also hits this, guess I was just getting lucky.

[Bauxitedev]

I'm getting timeouts repeatedly now. I can hardly run any code in the playground, even just basic hello worlds don't run anymore.

[shepmaster]

(responding to #762 (comment))

To fix this, the timeout duration could be increased from 10 seconds to a higher value.

I'm not well-versed in queuing theory, but wouldn't this make it worse or otherwise not help? If we assume a consistent rate of requests and we increase the timeout, then more compilations will be occurring concurrently, fighting each other for the same resources, slowing each other down for a longer period.

I also expect that at some inflection point, people will assume "oh, something broke" while waiting for a request and will resubmit the request, exacerbating the problem as now that one user has doubled their load. Even preventing a second submit in the UI is problematic as people can reload the page and retry, which is even a tiny bit more expensive.

Alternatively, cargo build and cargo run could have different timeouts, for example 15 seconds for compilation and 2 seconds for running.

This is perhaps something that we should have otherwise, but I also don't see how it would fix the problem. Compilation and running both contend for the same CPU time slice.

[shepmaster]

For a picture, here is "Timeouts vs Load average":

[shepmaster]

And for the last 7 days