rustdoc-ui is failing for my local (stage1) build (was: "I cannot tell which test suites the CI will run")

[pnkfelix]

I saw some local failures when I ran x.py test src/test/rustdoc-ui, running against an unmodified checkout.

From this, I infer that our CI is not testing that directory, at least not on a x86_64 Linux host.

There are other test suites in a somewhat similar situation, such as src/test/debuginfo, where I think we are not testing against lldb anywhere in CI (though I do think those tests are exercised against gdb, so that's not quite as bad).

Anyway, if there is information about which test suites are exercised by the CI and which ones are not, I could not immediately find it.

And it probably would be good to have some way to double-check such things. E.g. a README in each test suite directory that includes some section that says if it not run by CI (and, maybe, for extra credit, if it is run by CI, then list the CI configs that will run that test suite. And then have the bors leave a comment when the entries go stale -- I don't think such a scenario warrants blocking PRs, but nagging comments seem reasonable).

[pnkfelix]

@jyn514 reports to me that the CI definitely is running tests against src/test/rustdoc-ui

The reason I don't see rustdoc-ui in the CI scripts is that the run of that test suite is part of the runs that are implied by running x.py test with no filters on the test suite. @jyn514 advised me to find that spot in the CI scripts by grepping for "--exclude src/tools/tidy`, which pointed me here:

rust/src/ci/docker/host-x86_64/x86_64-gnu-llvm-10/Dockerfile

Lines 35 to 52 in 8daad74

	ENV SCRIPT python2.7 ../x.py --stage 2 test --exclude src/tools/tidy && \
	# Run the `mir-opt` tests again but this time for a 32-bit target.
	# This enforces that tests using `// EMIT_MIR_FOR_EACH_BIT_WIDTH` have
	# both 32-bit and 64-bit outputs updated by the PR author, before
	# the PR is approved and tested for merging.
	# It will also detect tests lacking `// EMIT_MIR_FOR_EACH_BIT_WIDTH`,
	# despite having different output on 32-bit vs 64-bit targets.
	python2.7 ../x.py --stage 2 test src/test/mir-opt \
	--host='' --target=i686-unknown-linux-gnu && \
	# Run the UI test suite again, but in `--pass=check` mode
	#
	# This is intended to make sure that both `--pass=check` continues to
	# work.
	#
	python2.7 ../x.py --stage 2 test src/test/ui --pass=check \
	--host='' --target=i686-unknown-linux-gnu && \
	# Run tidy at the very end, after all the other tests.
	python2.7 ../x.py --stage 2 test src/tools/tidy

So maybe my request for automatically-maintained info in a per-test-suite README is overkill. (But at the same time, it doesn't seem absurd for us to provide it.)

[jyn514]

Can you please post what failure you're seeing?

[Mark-Simulacrum]

I've long wanted us to collect and track which test suites are run on each builder, but we don't currently do that. It would not be too hard to start, but definitely represents a good amount of time investment on the design and implementation.

[pnkfelix]

Can you please post what failure you're seeing?

Pasted in: https://gist.github.com/pnkfelix/99e0203960c48994c2bf93064b0344c2

Many of the failures seem to be caused by the actual output omitting the expected text "compile fail". @jyn514 hypothesized on Zulip that maybe the test suite has a bug that is causing the wrong binary to be invoked.

[pnkfelix]

bisection points to PR #84863, but @jyn514 has privately indicated that PR #84863 is most likely not itself incorrect.

Rather it is more likely that PR #84863 has exposed a latent bug in our test suite.

[ehuss]

The issue is that the default for ./x.py test is --stage 1, which means that rustdoc will be built with the beta compiler. rustdoc is using the test framework from beta (see conditional here), which does not have the same output.

I'm not sure how that is usually handled. Maybe add // ignore-stage1 on those tests? However, someone will need to remember to remove those.

I'm kinda surprised more people haven't reported this, I would assume some people are running these tests locally?

EDIT: Oh, and since bootstrap will be bumped soon, it's probably not worth making any changes for these specific errors.

[jyn514]

I'm kinda surprised more people haven't reported this, I would assume some people are running these tests locally?

Almost everyone on the rustdoc team uses download-rustc, which compiles and tests stage 2. So the only people running into this are compiler devs testing rustdoc.

[ExpHP]

I'm kinda surprised more people haven't reported this, I would assume some people are running these tests locally?

I am surprised as well! As a data point, I ran into this yesterday naively running ./x.py test without any further arguments. (I am a very infrequent contributor and was just doing this from memory, and I wanted to check my WSL environment before starting development).