Download a pre-compiled bootstrap from CI

[jyn514]

Currently, we unconditionally recompile bootstrap on all builds, like libstd. In most cases, this is unnecessary: bootstrap is rarely changed and it's confusing that we have to build a whole rust program before processing x.py --help. Additionally, it means that when we add the new shell script entrypoints, we'd have to replicate all of bootstrap.py's logic for finding config.toml and the build directory. If we can download bootstrap through rustup, it will greatly simplify all that logic.

Most of this work has already been done in #98483. The work left is to actually download the generated artifact in bootstrap.py.

cc @dvtkrlbs

Mentoring instructions:

1. Add a new download_or_build_bootstrap function to src/bootstrap/bootstrap.py.
2. If there are changes to src/bootstrap (you can find these with git diff-index --quiet HEAD -- src/bootstrap), then always rebuild it from source.
3. Otherwise, download it from ci-artifacts.rust-lang.org, as configured by stage0.json - you may have to modify the Stage0Toolchain python object to load this configuration.

Helps with #94829.

[dvtkrlbs]

@rustbot claim

[est31]

bootstrap is rarely changed

I'm not sure I agree. According to git log --pretty=format:"%ad" --author "bors@rust-lang.org" src/bootstrap/ | rg 2022 | wc -l there have been 73 PRs merged this year that changed something in bootstrap. If we assume 211 days since Jan 1, there has been a change every 2.9 days. Compare that to 16 updates of src/llvm-project in 2022 (obtained with a similar command, but bors commits don't show up in llvm-project).

But it's still bearable I think. Definitely worth the cost of hosting a few binaries.

[jyn514]

@est31 I'd guess probably a good half of those are related to #94829. We should be seeing less churn once that gets closed out.

A fair number of the rest are related to #44293, which has also been fixed and won't have more PRs in the future.

Bootstrap is definitely changing more than LLVM, but I don't know that it's a great metric to compare to; I think bootstrap is a failiy small download compared to everything else.

[Mark-Simulacrum]

It's also true that we could tune the divide here over time - for example, splitting bootstrap into configuration+ flag parsing and the inner bits which actually run things.

Ultimately, the primary goal with downloading it is to reduce the amount of non-Rust code we maintain, and we need a pretty thin shim to parse flags, config, and download bootstrap compiler to proceed from there.

But yes, we should make sure the binary is not unexpectedly large on download - hopefully just a few megabytes.

[jyn514]

@dvtkrlbs 👋 it's been a while - are you still planning to work on this?

[est31]

FTR if you run the command from above then you get 13 PRs that changed bootstrap in 2023, which gives a 11 days per PR average rate, which is close to the LLVM change rate quoted above. And with serious build time increases proposed, like the 20%+ increase proposed in #108083, this makes more sense than it used to IMO.

[dvtkrlbs]

@dvtkrlbs 👋 it's been a while - are you still planning to work on this?

sadly no I am removing myself from the issue.