`incremental = true` causes 4-5x build time regression in generated rustc build times

[RalfJung]

I am running rm build/*/stage* -rf && ./x.py build src/libstd in two different situations: With incremental = true and with incremental = false. All the other settings in config.toml are commented out. This is using 1731f0a.

With incremental = false, the build step "Building stage1 std artifacts" takes 46s.
With incremental = true, the same step takes 4m 46s.

These are compiled using the stage0 compiler artifacts -- seems like those become way slower when incremental build is enabled?

(I am currently doing this experiment again just to be sure.)

[oli-obk]

cc @michaelwoerister

[RalfJung]

I have reproduced this. I also refined the command to rm build/*/stage* -rf && ./x.py build src/libstd --stage 1 to take less long. ;)

incremental = false:

Building stage1 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
   Compiling cc v1.0.17
   Compiling core v0.0.0 (file:///home/r/src/rust/rustc/src/libcore)
   Compiling build_helper v0.1.0 (file:///home/r/src/rust/rustc/src/build_helper)
   Compiling unwind v0.0.0 (file:///home/r/src/rust/rustc/src/libunwind)
   Compiling compiler_builtins v0.0.0 (file:///home/r/src/rust/rustc/src/rustc/compiler_builtins_shim)
   Compiling cmake v0.1.31
   Compiling alloc_jemalloc v0.0.0 (file:///home/r/src/rust/rustc/src/liballoc_jemalloc)
   Compiling std v0.0.0 (file:///home/r/src/rust/rustc/src/libstd)
   Compiling rustc_msan v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_msan)
   Compiling rustc_tsan v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_tsan)
   Compiling rustc_lsan v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_lsan)
   Compiling rustc_asan v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_asan)
   Compiling libc v0.0.0 (file:///home/r/src/rust/rustc/src/rustc/libc_shim)
   Compiling alloc v0.0.0 (file:///home/r/src/rust/rustc/src/liballoc)
   Compiling std_unicode v0.0.0 (file:///home/r/src/rust/rustc/src/libstd_unicode)
   Compiling alloc_system v0.0.0 (file:///home/r/src/rust/rustc/src/liballoc_system)
   Compiling panic_abort v0.0.0 (file:///home/r/src/rust/rustc/src/libpanic_abort)
   Compiling panic_unwind v0.0.0 (file:///home/r/src/rust/rustc/src/libpanic_unwind)
    Finished release [optimized] target(s) in 48.04s

incremental = true:

Building stage1 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
   Compiling cc v1.0.17
   Compiling core v0.0.0 (file:///home/r/src/rust/rustc/src/libcore)
   Compiling build_helper v0.1.0 (file:///home/r/src/rust/rustc/src/build_helper)
   Compiling unwind v0.0.0 (file:///home/r/src/rust/rustc/src/libunwind)
   Compiling compiler_builtins v0.0.0 (file:///home/r/src/rust/rustc/src/rustc/compiler_builtins_shim)
   Compiling cmake v0.1.31
   Compiling alloc_jemalloc v0.0.0 (file:///home/r/src/rust/rustc/src/liballoc_jemalloc)
   Compiling std v0.0.0 (file:///home/r/src/rust/rustc/src/libstd)
   Compiling rustc_asan v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_asan)
   Compiling rustc_tsan v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_tsan)
   Compiling rustc_lsan v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_lsan)
   Compiling rustc_msan v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_msan)
   Compiling libc v0.0.0 (file:///home/r/src/rust/rustc/src/rustc/libc_shim)
   Compiling alloc v0.0.0 (file:///home/r/src/rust/rustc/src/liballoc)
   Compiling std_unicode v0.0.0 (file:///home/r/src/rust/rustc/src/libstd_unicode)
   Compiling alloc_system v0.0.0 (file:///home/r/src/rust/rustc/src/liballoc_system)
   Compiling panic_abort v0.0.0 (file:///home/r/src/rust/rustc/src/libpanic_abort)
   Compiling panic_unwind v0.0.0 (file:///home/r/src/rust/rustc/src/libpanic_unwind)
    Finished release [optimized] target(s) in 4m 51s

[RalfJung]

This does not just affect libstd. If I can trust my backscroll here, a "stage1 compiler artifacts" build takes 25m 20s with incremental = true, whereas it just takes 8m 06s with incremental = false.

[michaelwoerister]

Yes, that's a known issue, unfortunately. #52309 will hopefully solve this in the medium term.

[alexcrichton]

With incremental ThinLTO now implemented, I think this is largely "fixed enough"