Overall cycle time has regressed ~25% from 2hrs to 2.5hrs

[alexcrichton]

Over the past few weeks it looks like our overall cycle time has regressed from ~2hrs to ~2.5 hours (roughly). The main contributing factors to this seem to be:

• OSX has regressed - I don't know how to easily solve this other than identifying the regression here. More discussion on the issue hopefully.
• Doubled-up dist builders are taking much longer - This is an unfortunate consequence of rustbuild: Add support for compiling Cargo  #39917. Compiling Cargo takes ~10 minutes so that's an extra 20 minutes right off the bat b/c we double-up dist builders. Furthermore compiling Cargo also requires a stage1 x86_64-unknown-linux-gnu compiler, which previously wasn't needed. That means that the doubled up dist builders are compiling one more rustc. This means that instantly the builders are taking an extra ~40-50 minutes.

AppVeyor in general looks pretty steady around or under 2 hrs, and all other builds are steadily under 2 hours as well. Note that 2 hours I'm taking as the "hard limit" here because that's how much time our absolute slowest builder, android, takes. I don't know how to optimize android any further so my goal has been to keep everything faster than Android.

I don't know how to solve the doubled up dist builder problem other than not doubling up dist builders. That would require an extra 8 builders on Travis. Currently we're using 33/45 slots. This is an unexpected increase in the number of builders so only 4 extra isn't great but it seems better than requiring all PRs to take longer.

Note that I'm considering this somewhat more urgent due to our desire to land the RLS in the distribution in the same manner as Cargo. Compiling the RLS takes ~10 minutes which would regress the doubled-up builders another 20 minutes, which is likely too unacceptable.

[brson]

Just options:

• Split the dist builders back out and eat up 8 additional jobs
• Split the dist builders back out and buy more capacity
• Disable parts of the test suite on macs
• Buy upgrades to the mac hardware
• Buy upgrades to the linux hardware
• Split the builds into multiple stages to avoid the Travis timeout
• Split the builds into multiple stages to avoid building redundant Linux hosts
• Move the cargo/rls build back out of tree to some other build process
• Move macs back off of travis onto macstadium directly so we have more control
• Move macs back into the moco datacenter
• Move the linux bots back off of travis onto AWS

ISTM that macs are the immediate problem. On linux we have the option of just splitting out the builds in the short-term.

We might consider what it would mean for the 'extended' build to be a separate build process while still maintaining the advantages of a single codebase and build system. The way we have it set up now where we build the toolchain, then use the toolchain to build the tools, feels right to me though.

Do the 'check' configurations run all the way through stage2 and check cargo? We could make the mac 'check' configurations not do extended builds for now, under the assumption nothing too horrible is going to go wrong (cargo and rls do have their own CI after all).

[alexcrichton]

#40998 landed, osx is tracked by #40802, closing in favor of that

[ishitatsuyuki]

I'm aware that this is closed, but have you considered making it build extended for a nightly cron trigger? That way, the tests which runs many times a day would be faster.