Tracking Issue for consolidating command invocations in bootstrap

[Kobzol]

This is a tracking issue for monitoring the work of improving how we deal with command invocations in bootstrap (proposed here).

Currently, bootstrap executes commands on many places, often in arbitrary ways. This makes it harder to debug, profile and test the command invocations executed by it. The goal of this refactoring is to make sure that all external command invocations use the same API (a custom command wrapper) and that they go through a centralized location.

Intermediate steps:

• Introduce a command wrapper abstraction
  • Centralize command running in boostrap (part one) #116581
• Port all existing simple command usages that have access to &Build(er) to BootstrapCommand
  • Bootstrap command refactoring: refactor BootstrapCommand (step 1) #126731
  • Bootstrap command refactoring: port more Command usages to BootstrapCmd (step 2) #126822
  • Bootstrap command refactoring: consolidate output modes (step 3) #127120
• Modify BootstrapCmd so that it stores Command instead of &mut Command and remove all calls to BootstrapCmd::from by changing Command::new to BootstrapCmd::new
  • Bootstrap command refactoring: port more Command usages to BootstrapCmd (step 2) #126822
• Consolidate how command output modes work (described here)
  • Bootstrap command refactoring: consolidate output modes (step 3) #127120
• Make it easier to use the command API in a fluent way, and mark methods with #[must_use]
  • Bootstrap command refactoring: quality-of-life improvements (step 4) #127321
• Add #[track_caller] to command execution functions to make debugging easier.
  • Bootstrap command refactoring: improve debuggability (step 5) #127450
• Implement drop bombs to check that each command has been executed at least once.
  • Bootstrap command refactoring: improve debuggability (step 5) #127450
• Refactor the rest of command executions not currently using BootstrapCmd that can access Builder to use the correct output and failure modes. This will include passing Builder to additional places.
  • Bootstrap command refactoring: port remaining commands with access to Build (step 6) #127680
• Make the API for getting a stdout of a command nicer.
  • Bootstrap command refactoring: make command output API more bulletproof (step 7) #127799
• Add a check that asserts that you don't try to get stdout/stderr of a command that does not capture stdout/stderr.
  • Bootstrap command refactoring: make command output API more bulletproof (step 7) #127799
• Refactor how flags are passed to Config.
  • Slightly refactor Flags in bootstrap #128878
• Remove access to inner command of BootstrapCommand.
• Handle the most complex cases, such as output streaming.
• Handle the co-operation of Builder, Build, Config and command context. There are some fields and logic used during command execution that are distributed amongst Builder/Build/Config, so it will require some refactoring to make it work if the execution will happen on a separate place (in the command context).
• Refactor the rest of commands that cannot access builder (e.g. Config::parse) by introducing a new command context that will be passed to these places, and then stored in Builder. Move certain fields (such as fail_fast) from Builder to the command context.
• Refactor logging of commands, so that it is either logged to a file or printed in a nice hierarchical way that cooperates with the Step debug hierarchical output.
• Implement profiling of commands (add command durations to the command log, print a log of slowest commands and their execution counts at the end of bootstrap execution, perhaps store command executions to metrics.json).
• Implement caching of commands.
• Implement testing of commands through snapshot tests/mocking.