Some tests are incorrectly skipped for macOS CI

[shepmaster]

CI specifies these two jobs for x86_64-apple-darwin:

rust/src/ci/github-actions/ci.yml

Lines 528 to 544 in 341efb1

	- name: x86_64-apple-1
	env: &env-x86_64-apple-tests
	SCRIPT: ./x.py --stage 2 test --skip tests/ui --skip tests/rustdoc --skip tests/run-make-fulldeps
	RUST_CONFIGURE_ARGS: --build=x86_64-apple-darwin --enable-sanitizers --enable-profiler --set rust.jemalloc --set llvm.ninja=false
	RUSTC_RETRY_LINKER_ON_SEGFAULT: 1
	MACOSX_DEPLOYMENT_TARGET: 10.12
	MACOSX_STD_DEPLOYMENT_TARGET: 10.12
	NO_LLVM_ASSERTIONS: 1
	NO_DEBUG_ASSERTIONS: 1
	NO_OVERFLOW_CHECKS: 1
	<<: *job-macos-xl
	- name: x86_64-apple-2
	env:
	SCRIPT: ./x.py --stage 2 test tests/ui tests/rustdoc tests/run-make-fulldeps
	<<: *env-x86_64-apple-tests
	<<: *job-macos-xl

Note these two lines (with padding I've added):

./x.py --stage 2 test --skip tests/ui --skip tests/rustdoc --skip tests/run-make-fulldeps 
./x.py --stage 2 test        tests/ui        tests/rustdoc        tests/run-make-fulldeps 

It appears that the intent was for the first job to run a subset of the tests and the second job to run the complementary subset. However, it seems that --skip performs a substring match while the list of directories is an exact match. This means that the tests in rustdoc-js, rustdoc-json, and rustdoc-ui are never run on macOS CI — they are excluded by --skip and never explicitly added back.

Surprisingly, it seems that tests in rustdoc-js-std are run. That seems like a secondary bug as it isn't behaving the same.

[shepmaster]

My suggestion would be to have another flag that performs the same substring match. For example, using --only as the option:

./x.py --stage 2 test --skip tests/ui --skip tests/rustdoc --skip tests/run-make-fulldeps 
./x.py --stage 2 test --only tests/ui --only tests/rustdoc --only tests/run-make-fulldeps 

[onur-ozkan]

If you don't want --skip to match as a SUBSTRING, you can use the --exact flag with x.

e.g., x --stage 2 tests/rustdoc-js --skip tests/rustdoc --force-rerun -- --exact

Does this help?

[madsmtm]

@rustbot label O-macos

[onur-ozkan]

If you don't want --skip to match as a SUBSTRING, you can use the --exact flag with x.

e.g., x --stage 2 tests/rustdoc-js --skip tests/rustdoc --force-rerun -- --exact

Does this help?

This should work fine, feel free to re-open the issue if it doesn't help.

[madsmtm]

We need to actually do this in our CI IMO before the issue should be closed.