Skip to content

Add initial version of snapshot tests to bootstrap #142431

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

Kobzol
Copy link
Contributor

@Kobzol Kobzol commented Jun 12, 2025

When making any changes to bootstrap (steps), it is very difficult to realize how does it affect various common bootstrap commands, and if everything still works as we expect it to. We are far away from having actual end-to-end tests, but what we could at least do is have a way of testing what steps does bootstrap execute in dry run mode. Now, we already have something like this in src/bootstrap/src/core/builder/tests.rs, however that is quite limited, because it only checks executed steps for a specific impl of Step and it does not consider step order.

Recently, when working on what I thought was one of the simplest possible step untanglings in bootstrap (#142357), I ran into errors in tests that were quite hard to debug. Partly also because the current staging test diffs are multiline and use Debug output, so it's quite difficult for me to make sense of them.

In this PR, I introduce insta, which allows writing snapshot tests in a very simple way. With it, I want to allow writing tests that will clearly show us what is going on during bootstrap execution, and then write golden tests for build/check/test stage 0/1/2 for compiler/std/tools etc., to make sure that we don't regress something, and also to help with #t-infra/bootstrap > Proposal to cleanup stages and steps after the redesign, to help avoid a situation where we would (again) have to make a flurry of staging changes because of unexpected consequences.

In the snapshot tests, we currently render the build of rustc, std and LLVM. Currently I render the executed steps using downcasting, which is not super pretty, but it allows us to make the test rendering localized in one place, and it's IMO enough for now.

I implemented only a single test using the new machinery. Maybe if you take a look at it, you will understand why 😆 Bootstrap currently does some peculiar things, such as running a stage 0 std step (even though stage 0 std no longer exists) and running the Rustc stage 0 -> 1 step twice, once with a single crates, once with all rustc crates. So I think that even with this single step, there will be a bunch of things to fix in the near future...

The way we currently prepare the Config test fixtures is far from ideal, this is something I think @Shourya742 could work on as a part of their GSoC project (remove as much command execution from Config construction as possible, actually run bootstrap on a temporary directory instead of running it on the rustc checkout, create a Builder-like API for creating the Config test fixtures).

r? @jieyouxu

@rustbot rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Jun 12, 2025
@rust-log-analyzer
Copy link
Collaborator

The job x86_64-gnu-tools failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)
[TIMING] core::build_steps::test::Crate { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu, forced_compiler: true }, target: x86_64-unknown-linux-gnu, mode: Std, crates: ["std"] } -- 69.430
Build completed successfully in 0:01:13
+ head -n 1 /tmp/browser-ui-test.version
+ npm install browser-ui-test@0.20.6 --unsafe-perm=true
npm ERR! code E504
npm ERR! 504 Gateway Timeout - GET https://registry.npmjs.org/@puppeteer%2fbrowsers

npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2025-06-12T18_48_13_817Z-debug-0.log
  local time: Thu Jun 12 18:50:11 UTC 2025
  network time: Thu, 12 Jun 2025 18:50:11 GMT
##[error]Process completed with exit code 1.
Post job cleanup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants