Redesign bootstrap stages

[jyn514]

Proposal

Make several breaking changes to how bootstrapping works (discussed in further detail below). See https://jyn.dev/2023/01/12/Bootstrapping-Rust-in-2023.html for greater detail and https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/Stage.20numbering.20for.20tools/near/326378168, https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/blog.20post/near/322763929 for past discussion.

Here is a high-level summary, although I highly recommend reading at least the blog post:

• Get rid of build --stage 0 std. Build rustc with beta std in stage0 and the in-tree std at all other times. Build std unconditionally with the in-tree rustc (either stage 1 or 2, never 0).
• Enable download-rustc = "if-unchanged" for library and compiler contributors, so that building std from source doesn't require first building the compiler.
  • Fix the existing bugs (I've been working on this the past 2 weeks with great success): Follow-up work for download-rustc rust#81930
  • Change download-rustc = "if-unchanged" to only rebuild std if std is modified, not both rustc and std
  • When download-rustc = "if-unchanged" is enabled, change x check std to use a downloaded compiler if rustc hasn't been modified.
  • The behavior for x check std when rustc has been modified is under active discussion.
• Require profile in config.toml, so that we can distinguish distros building from source from other contributors. There are a bunch of mitigating factors that help if profile isn't required, like how ./configure sets profile = user by default and if-unchanged will still build rustc from source on changes, but overall it seems better to have people explicitly say what they want to do with bootstrap.
  • Add a new profile = "none" option in case none of the current profiles are a good fit.
• Add a new --target-sysroot <bootstrap|dev|dist> flag that is recommended over --stage. They are not aliases, they will do different things. --stage will remain supported for at least ~3 months but I would like to remove it eventually. I am open to bikeshedding on the name of the flag, but I feel strongly that it should be a new flag, not silently change the behavior of --stage.
  • See below for a diagram outlining the differences. See https://jyn.dev/2023/01/12/Bootstrapping-Rust-in-2023.html#what-can-we-do-instead for a detailed discussion.

I understand that these are large breaking changes and it may be unclear how to use the new flags. To ease the transition, I am willing to put up a prototype branch with the changes to add --target-sysroot and remove stage 0 std, so that people can try it out before it lands. I am not going to keep it up to date with latest master, but it should be enough to run various commands and get a feel for how it works.

Assuming this is accepted, I am also planning to write a blog post including the above diagram and some examples of how to transition to the new flags. See https://jyn.dev/2023/01/12/Bootstrapping-Rust-in-2023.html#what-can-we-do-instead for existing examples of how to transition.

Motivation

Again, I highly recommend reading the blog post for more detailed discussion.

Stages are very confusing. The main stumbling block is that people expect build --stage N foo to mean "create the foo that will end up in build/stageN/", but it actually means "use the compiler in build/stageN to build foo". This is the source of a lot of off-by-one errors. --target-sysroot will avoid this problem by both changing the behavior to match people's expectations, and changing the name to be less ambiguous.

test and build are inconsistent in several places (and build is inconsistent even with itself when it comes to tools; see https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/Stage.20numbering.20for.20tools/near/298189698 for further discussion). In particular, while build --stage 1 rustc and test --stage 1 rustc_data_structures both build the compiler twice, test --stage 1 tests/ui only builds the compiler once. --target-sysroot dev will unify all these as "build the compiler once (and possibly run it on a test file)".

Building std with two different compilers, and mixing and matching beta rustc with nightly std, is unlike any other program. Switching std to be built with a single toolchain, and rustc to be built by a unified toolchain rather than mixing and matching, will help make both rustc and std less special:

• std no longer needs cfg(bootstrap) at all, making it much easier to add new lang items and modify builtin macros like format_args
• rustc can "just" be built with a pinned beta or rebuilt with itself. I hope that over time this will let bootstrap itself become a lot simpler.

I've heard concerns in the past that this will "just move cfg(bootstrap) to the compiler". I think that won't happen because rustc isn't tied to std internals the way std is tied to rustc internals. See https://jyn.dev/2023/01/12/Bootstrapping-Rust-in-2023.html#but-that-just-moves-the-cfgbootstrap-to-the-compiler for more discussion.

Mentors or Reviewers

@RalfJung and @Mark-Simulacrum on the design and implementation, @nnethercote and @matklad on the design

Process

The main points of the Major Change Process are as follows:

• File an issue describing the proposal.
• A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Comments

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

[rustbot]

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

cc @rust-lang/compiler @rust-lang/compiler-contributors

[WaffleLapkin]

@rustbot second

[apiraino]

@rustbot label -final-comment-period +major-change-accepted