Add initial version of snapshot tests to bootstrap #142431
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When making any changes to bootstrap (steps), it is very difficult to realize how does it affect various common bootstrap commands, and if everything still works as we expect it to. We are far away from having actual end-to-end tests, but what we could at least do is have a way of testing what steps does bootstrap execute in dry run mode. Now, we already have something like this in
src/bootstrap/src/core/builder/tests.rs
, however that is quite limited, because it only checks executed steps for a specific impl ofStep
and it does not consider step order.Recently, when working on what I thought was one of the simplest possible step untanglings in bootstrap (#142357), I ran into errors in tests that were quite hard to debug. Partly also because the current staging test diffs are multiline and use
Debug
output, so it's quite difficult for me to make sense of them.In this PR, I introduce
insta
, which allows writing snapshot tests in a very simple way. With it, I want to allow writing tests that will clearly show us what is going on during bootstrap execution, and then write golden tests forbuild/check/test
stage0/1/2
for compiler/std/tools etc., to make sure that we don't regress something, and also to help with #t-infra/bootstrap > Proposal to cleanup stages and steps after the redesign, to help avoid a situation where we would (again) have to make a flurry of staging changes because of unexpected consequences.In the snapshot tests, we currently render the build of rustc, std and LLVM. Currently I render the executed steps using downcasting, which is not super pretty, but it allows us to make the test rendering localized in one place, and it's IMO enough for now.
I implemented only a single test using the new machinery. Maybe if you take a look at it, you will understand why 😆 Bootstrap currently does some peculiar things, such as running a stage 0 std step (even though stage 0 std no longer exists) and running the Rustc stage 0 -> 1 step twice, once with a single crates, once with all rustc crates. So I think that even with this single step, there will be a bunch of things to fix in the near future...
The way we currently prepare the Config test fixtures is far from ideal, this is something I think @Shourya742 could work on as a part of their GSoC project (remove as much command execution from Config construction as possible, actually run bootstrap on a temporary directory instead of running it on the rustc checkout, create a Builder-like API for creating the Config test fixtures).
r? @jieyouxu