bootstrap: `check compiler` stages are off compared to `build` stages

[RalfJung]

@Kobzol and me have been discussing what the stage numbers in bootstrap should be "from first principles". Zulip thread

Here's a little diagram showing the core dependency loop: (miri is a stand-in for any tool depending on rustc crates, e.g. clippy, RA)

The "build" edge means "compiler is used to build that crate", a "dep" edge means "crate A is needed to build crate B". Miri of course also has a "dep" edge from the library, but I omitted that as it exists transitively.

Bootstrap needs to cut this loop, and since the recent stage redesign it does this as follows:

src

This is how ./x build already behaves today: build $thing --stage $N will build "$thing $N" from that diagram.

We can annotate the original diagram with staging offsets on the arrows to cut out the redundancy:

However, it turns out ./x check compiler currently does not follow the same staging system as ./x build. Running ./x check compiler --stage 1 will in fact do the equivalent of ./x build compiler --stage 2! However, ./x check library --stage 1 does match ./x build library --stage 1 as expected. miri behaves like compiler, i.e., it is also off-by-one in check.

I don't know why check is off-by-1 from build for everything except for library, but it'd be great to make this consistent. :) Checking the stage 1 compiler should be using the stage 0 (beta) compiler to check the in-tree sources.

Cc @rust-lang/bootstrap

Bonus content

Here's a version of the above diagram with all "build" edges in it, with transitive "dep" edges made explicit, and with cargo as well as a stand-in for a tool that does not depend on compiler crates.

src

Key invariants:

• Every node has exactly one incoming "build" edge, which always comes from compiler.
• Every loop must have at least one "+1" on an edge.
• If we have "compiler --build--> A" and "compiler --build--> B --dep--> A", then the sum of the stage bumps along both paths must be the same. This ensures a crate and its dependencies get built with the same compiler.

I strongly feel that we want the process to be described by some form of diagram like that, to make it all consistent. The degrees of freedom, given the above invariants, are few:

• In the one loop of the diagram, we could put the "+1" on the other edge. But then the sequence of steps that bootstrap would perform for a stage N compiler build would be: compiler 1, library 2, compiler 2, ... There would be no "library 1". That seems very strange.
• "cargo" could avoid the "+1" on both of its incoming edges. But that would mean a stage 1 cargo would be built with stage 1 rustc, i.e. build --stage 1 cargo would have to first do a full rustc build (or download the prebuilt binary) -- that seems entirely unnecessary, I see no benefits from doing this.

Given this design I think we can set stage 1 to be the default stage for almost every command (and treat the remaining commands where that does not behave well as bugs). There's no reason to have different default stages for test vs build vs check, or for different profiles, which hopefully means a bunch of complexity can be removed from bootstrap. :)

FWIW, --stage 0 does not really ever make sense as a concept in this world. IMO that should just always be an error, but @Kobzol suggested that'd break too many things for tools that don't depend on rustc, so maybe it should just be an alias for --stage 1 (with a warning that we could eventually make an error). We could also shift everything by 1 so that stage 0 becomes the first stage bootstrap builds, but then we'd need to consider the downloaded rustc+std to be "stage -1" and I don't think we want to go there.

[onur-ozkan]

Thanks for putting a lot of effort on this. :)

To make put it clear, with this design, if someone want to build a simple tool like src/tools/remote-test-client with the precompiled stage0 compiler, they will have to pass --stage 1 flag to the build command, right? In different perspective, bootstrap will use stageN-1 compiler to build stageN artifact, which can be quite confusing for some commands, I think?

It would be great to also consider renaming the --stage flag in this design, as it can be quite confusing in certain cases (like the one I just mentioned above). I think it's reasonable for users to expect that "stageN (as passed to the command) means using the stageN compiler to build things"; maybe we should come up with a better name for it (there are plenty of them already suggested in the past).

Given this design I think we can set stage 1 to be the default stage for almost every command (and treat the remaining commands where that does not behave well as #78717). There's no reason to have different default stages for test vs build vs check, or for different profiles, which hopefully means a bunch of complexity can be removed from bootstrap. :)

Yes, there are a bunch of options in bootstrap.toml (like check-stage, build-stage, test-stage, etc.) that we can remove after this.

FWIW, --stage 0 does not really ever make sense as a concept in this world. IMO that should just always be an error, but @Kobzol suggested that'd break too many things for tools that don't depend on rustc, so maybe it should just be an alias for --stage 1 (with a warning that we could eventually make an error).

I am not sure if that's a bad thing. If they break, updating a bunch of commands/flags should be an easy fix unless I am missing something. Also, if we rename --stage flag, everything will break anyway.

[Kobzol]

To make put it clear, with this design, if someone want to build a simple tool like src/tools/remote-test-client with the precompiled stage0 compiler, they will have to pass --stage 1 flag to the build command, right?

It would definitely not be needed to do this, as --stage 1 would be the default anyway. We can either make bootstrap tools be super consistent with the rest of the other steps, or we can just keep the status quo and hard-code building them with the stage 0 rustc. In any case, default x build/check on a bootstrap tool should always do the right thing, without users having to pass stage explicitly (#t-infra/bootstrap > Stage numbering no longer agrees with bootstrap loop @ 💬).

In different perspective, bootstrap will use stageN-1 compiler to build stageN artifact, which can be quite confusing for some commands, I think?

This happens for most steps, I think? Except for the standard library. You build the stage 1 compiler using the stage 0 compiler etc.

I think it's reasonable for users to expect that "stageN (as passed to the command) means using the stageN compiler to build things";

This interpretation probably isn't ideal after the redesign, as we have discussed in https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Stage.20numbering.20no.20longer.20agrees.20with.20bootstrap.20loop/with/523109521. I'll try to prepare also a version of the command tables with this approach so that we can compare it, but I think in the new design it's not the right fit.

[onur-ozkan]

It would definitely not be needed to do this, as --stage 1 would be the default anyway. We can either make bootstrap tools be super consistent with the rest of the other steps, or we can just keep the status quo and hard-code building them with the stage 0 rustc.

It's not about making or not making things. What I am saying is it can be confusing for people. That's why I suggest covering --stage renaming in this work.

This happens for most steps, I think? Except for the standard library. You build the stage 1 compiler using the stage 0 compiler etc.

There are many tools (e.g., here any tool that isn't is_unstable_tool = true) not doing it.

[Kobzol]

Yeah, as Ralf noted, bootstrap tools are a bit weird, as they don't have to even participare in any staging. And they don't participate in it even today, e.g. if you do x build opt-dist --stage 2, it still builds opt-dist using the stage 0 compiler. So we can just keep doing that.

Essentially, if we use the "new" staging terminology, we can do the following for bootstrap tools:

• Handle staging for them consistently with compiler/library. That means that the default stage is 1, --stage 0 is a no-op, --stage 1 builds with stage 0 rustc, --stage 2 builds with stage 1 rustc. This would be a breaking change in a way (although I suspect it wouldn't matter for most people), for added consistency and the ability to compile these tools using the in-tree compiler (but ~no one seems to need that, as we didn't support this before).
• We can continue doing what we did before, ignore staging for them completely, and always build them with stage 0 rustc.

[onur-ozkan]

Essentially, if we use the "new" staging terminology, we can do the following for bootstrap tools:

• Handle staging for them consistently with compiler/library. That means that the default stage is 1, --stage 0 is a no-op, --stage 1 builds with stage 0 rustc, --stage 2 builds with stage 1 rustc. This would be a breaking change in a way (although I suspect it wouldn't matter for most people), for added consistency and the ability to compile these tools using the in-tree compiler (but ~no one seems to need that, as we didn't support this before).
• We can continue doing what we did before, ignore staging for them completely, and always build them with stage 0 rustc.

The first option seems to be the only solution if we want consistency across every x command.

[RalfJung]

To make put it clear, with this design, if someone want to build a simple tool like src/tools/remote-test-client with the precompiled stage0 compiler, they will have to pass --stage 1 flag to the build command, right? In different perspective, bootstrap will use stageN-1 compiler to build stageN artifact, which can be quite confusing for some commands, I think?

It would be great to also consider renaming the --stage flag in this design, as it can be quite confusing in certain cases (like the one I just mentioned above). I think it's reasonable for users to expect that "stageN (as passed to the command) means using the stageN compiler to build things"; maybe we should come up with a better name for it (there are plenty of them already suggested in the past).

This presupposes that "stage" somehow implies "the sysroot used to build this". That was never its defining property, and I think a lot of the confusion comes from people trying to shove "stage" into such a mental model. As you will note, my summary above fully motivates and defines staging without ever describing it in terms of which toolchain it acts on or anything like that. Stages are just a numbering scheme applied to an unrolled dependency loop.

After systematically computing what a "stage" is using the method I laid out above, we can then observe the result and try to cast it in these kinds of terms. Personally I don't think that's a good idea, it obfuscates the underlying structure of the bootstrap dependency loop, but many people seem to be drawn to that way of thinking. If we do this, then under the old world (before the stage refactor, when library was the first thing that got built; here's a diagram), a "stage" indeed was consistently built with a single compiler. But with in the new world (when compiler is the first thing that gets built, and visualized in the diagrams above), this is no longer the case; it is now closer to "a stage is all the stuff that makes up a coherent toolchain". So e.g. the stage 1 compiler and stage 1 library together form a toolchain, and then one could add a stage 1 Miri, etc. --stage N means "build for the stage N toolchain", not "build with the stage N toolchain".

But in the end, most people should not pass --stage anyway; it's largely an internal concept.

So, I think there's no reason to rename anything. Also note that build and test already behave the way I describe above, since the stage refactor landed: build compiler --stage 1 and build library --stage 1 use different compilers to build the thing they are asked to build! So it's a bit too late to do the rename, too.

---

We can either make bootstrap tools be super consistent with the rest of the other steps, or we can just keep the status quo and hard-code building them with the stage 0 rustc.

The 2nd option here seems to be equivalent to saying "always pretend they are in stage 1", under the model described above. (Their dependency graph looks like that of cargo.)

--stage 0 is a no-op

Is there a good reason to make it a silent no-op? I feel like it should either be a hard error, or a "warning: you probably meant stage 1 instead", followed by pretending they wrote stage 1. The latter could make sense for a smoother transition from the status quo.

[onur-ozkan]

After systematically computing what a "stage" is using the method I laid out above, we can then observe the result and try to cast it in these kinds of terms. Personally I don't think that's a good idea, it obfuscates the underlying structure of the bootstrap dependency loop, but many people seem to be drawn to that way of thinking.

There will be more people falling into that way of thinking and I am quite sure about it.

So e.g. the stage 1 compiler and stage 1 library together form a toolchain, and then one could add a stage 1 Miri, etc. --stage N means "build for the stage N toolchain", not "build with the stage N toolchain".

If we make it more explicit (perhaps something like --target-stage or --for-stage), that would be enough to make it clear for anyone. It should not hurt the project in any way except for some transition time for core developers, which can be handled with a soft migration without removing the --stage flag directly.

[RalfJung]

That seems like entirely unnecessary churn to me. What is an example of someone / some workflow that would benefit from the rename? No rename was suggested with the stage redesign to my knowledge, despite that being the moment where --stage "changed its meaning" for all commands except check.

I'd rather not have to permanently use an unnecessarily long flag name just because some people misunderstood staging at some point.

The docs for --stage even start by saying "stage to build", which is still correct. That's what the flag was always meant to mean: "build the stage N version of this thing". It just happened to be the case that this was equivalent to "build this thing with the stage N-1 compiler".

FWIW the docs also say "stage 0 uses the bootstrap compiler" which seems to indicate that maybe we should start counting the stages at 0, rather than 1 (with bootstrap being stage 0) as in my writeup? Then build compiler --stage 0 would indeed still use the bootstrap compiler. build library --stage 0 obviously cannot do that though so the docs need an update either way. OTOH, according to this table, build and test already mostly behave like stage numbers start at 1, so it seems unnecessary to change that.

[Kobzol]

jyn suggested different CLI flags in their blog post a few years ago (https://jyn.dev/bootstrapping-rust-in-2023/#what-can-we-do-instead), that's kind of a "big rename". A smaller rename would be to simply rename --stage to --target-stage or something. I'm not convinced it's worth the churn at this point.