ui test with ICE does not fail

[RalfJung]

We are using this in Miri, and I just have the situation that one of our UI tests ICEs when I call it directly but the test is considered passing by compiletest. I have copy-pasted the command-line (as it gets printed when I make the test really fail, e.g. by introducing a syntax error) to make sure it is not some weird extra flag.

On a possibly related note, when I run the test binary directly (target/debug/deps/compiletest-edd6bd72083f7b79), all tests should fail because Miri complains about a missing shared library. Instead, only those tests with a non-empty stdout/stderr reference file fail, complaining that stdout/stderr was empty instead. This is particularly curious since stderr is not actually empty:

normalized stderr:


expected stderr:
a


diff of stderr:

-a
-

The actual stderr differed from the expected stderr.
Actual stderr saved to /tmp/compiletest9fDVl0/fn_box.stderr
To update references, run this command from build directory:
tests/run-pass/update-references.sh '/tmp/compiletest9fDVl0' 'fn_box.rs'

error: 1 errors occurred comparing output.
status: exit code: 127
command: "target/debug/miri" "tests/run-pass/fn_box.rs" "-L" "/tmp/compiletest9fDVl0" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-C" "prefer-dynamic" "-o" "/tmp/compiletest9fDVl0/fn_box.stage-id" "-Dwarnings" "-Dunused" "--edition" "2018" "-L" "/tmp/compiletest9fDVl0/fn_box.stage-id.aux" "-A" "unused"
stdout:
------------------------------------------

------------------------------------------
stderr:
------------------------------------------
target/debug/miri: error while loading shared libraries: librustc_driver-3f5e8a7f565acbfe.so: cannot open shared object file: No such file or directory

Compare "normalized stderr" and "stderr".

I have no idea how this is possible.

[RalfJung]

Okay, so the reason that "normalized stderr" and "stderr" are so different is about something with JSON stuff... if there is no --error-format, it tries to do something smart with JSON, and that can have really surprising consequences. Namely it looks like when rustc ICEs, that is not printed in JSON format even with --error-format json and hence json::extract_rendered just returns the empty string.

So, when there is a test that expects empty stderr, and that test ICEs, compiletest normalizes the stderr to empty and considers the test passed.

Cc @gnzlbg @alexcrichton

[RalfJung]

@laumann looks like runtest.rs in rustc master has check_if_test_should_compile that would catch cases like this. Any chance you could update the copy in this repo?

I am still not sure though if the behavior around normalization and JSON is intended. This all looks terribly fragile...

[RalfJung]

FWIW, it is only due to this bug that Miri currently looks green in the toolstate tracking. Actually, some of the tests fail with an ICE since rust-lang/rust#59500 landed.

[phansch]

I'm not sure if that works for Miri, but in Clippy I'm currently using // run-pass as a workaround. A test that can't compile will not be runnable, so compiletest will fail at that point because it can't find the compiled file. But yes, it would be good to fix the core issue here.

[RalfJung]

I guess Miri could do that, but we don't actually want to run these things so that seems like a waste of test time.

See #172 / rust-lang/rust#59769 for a possible fix.