Add caching layer to bootstrap

[Shourya742]

This PR adds a caching layer to the bootstrap command execution context. It is still a work in progress but introduces the initial infrastructure for it.

r? @Kobzol

[Kobzol]

Awesome! I haven't checked the code yet, but once you get it working, it would be great to see some benchmarks. Ideally of "no-op" commands, like running x check rustc after no change has been made to the compiler.

[Kobzol]

It seems like you misunderstood what I meant and went in a different direction, sorry :) I tried to clarify in comments.

[Kobzol]

I forgot that std::process::Command now provides a way to return all the inner state regarding arguments, env. variables, program and workdir. In that case, we don't need to store the cache key at all, and we can just reconstruct it from the inner std Command right before executing the bootstrap command.

[Kobzol]

Seems like some cache hits are being encountered, that's good. Later it would be nice to also record the durations of individual commands and then print some statistics at the end, which commands took the longest to execute, how many cache hits/misses they had etc.

[Kobzol]

Code looks good now, but before we merge it, I think that we should go through existing commands and make sure that it won't be problematic to cache them. I'll go through bootstrap tomorrow and check.

It would also be useful to add a comment on top of BootstrapCommand that explains that by default, command executions are cached if they have the same workdir/program/args/envs.

[Shourya742]

Code looks good now, but before we merge it, I think that we should go through existing commands and make sure that it won't be problematic to cache them. I'll go through bootstrap tomorrow and check.

It would also be useful to add a comment on top of BootstrapCommand that explains that by default, command executions are cached if they have the same workdir/program/args/envs.

I will add that.. added here: a5df236

[Kobzol]

Just to be sure, use do_not_cache on the command created in the bare_cargo function, to make sure that we don't ever cache cargo.

Also, it's a big edge case, but calling as_command_mut in ExecutionContext::start disables the caching of the command. Since our API allows executing the same command twice (although we don't currently do it anywhere, AFAIK), it could be cached in the first execution, and then not cached in the other executions, which is confusing. So it would be better if the execution context didn't call the function and just accessed the inner command directly. At the same time, I noticed that we have both exec.rs and execution_context.rs, which is a bit duplicated, both the command and the execution context belong together. So I would suggest moving the execution context to the exec.rs file (and remove execution_context.rs), which will also allow the context to access the inner std command without going through the as_command_mut function.

[Shourya742]

Oops

[rustbot]

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

[Kobzol]

Thank you! Please squash the commits and I'll approve the PR. We'll see how it works in practice.

[rustbot]

Some changes occurred to the CTFE / Miri interpreter

cc @rust-lang/miri

Some changes occurred in compiler/rustc_passes/src/check_attr.rs

cc @jdonszelmann

rustc_errors::translation was changed

cc @davidtwco, @compiler-errors, @TaKO8Ki

Some changes occurred in compiler/rustc_attr_parsing

cc @jdonszelmann

rustc_error_messages was changed

cc @davidtwco, @compiler-errors, @TaKO8Ki

Some changes occurred in compiler/rustc_codegen_ssa

cc @WaffleLapkin

rustdoc-json-types is a public (although nightly-only) API. If possible, consider changing src/librustdoc/json/conversions.rs; otherwise, make sure you bump the FORMAT_VERSION constant.

cc @CraftSpider, @aDotInTheVoid, @Enselic, @obi1kenobi

Some changes occurred in src/tools/rustfmt

cc @rust-lang/rustfmt

Some changes occurred in compiler/rustc_attr_data_structures

cc @jdonszelmann

Some changes occurred to the CTFE machinery

cc @RalfJung, @oli-obk, @lcnr

Some changes occurred in compiler/rustc_codegen_ssa/src/codegen_attrs.rs

cc @jdonszelmann

Some changes occurred in HTML/CSS/JS.

cc @GuillaumeGomez, @jsha, @lolbinarycat

Some changes occurred in src/tools/clippy

cc @rust-lang/clippy