`cargo miri test` is broken for doctests (Option 'sysroot' given more than once)

[RalfJung]

rust-lang/rust#122840 changed the way rustdoc passes arguments to rustc for doctest builds, which causes cargo miri test to break. The first symptom is output like

---- src/lib.rs - my_fn (line 37) stdout ----
error: Option 'sysroot' given more than once

Couldn't compile the test.

The deeper cause is that Miri does a lot of scanning and patching of rustc command-line arguments, and if those arguments are hidden in a file then everything breaks.

[RalfJung]

cargo miri test -v shows

[cargo-miri rustc inside rustdoc] going to run:
MIRI_BE_RUSTC="target" "/home/r/.rustup/toolchains/miri/bin/miri" "@/tmp/rustdoctestAMWaIS/rustdoc-cfgs" "--edition" "2018" "-o" "/tmp/rustdoctestyUJi8S/rust_out.miri" "--emit=metadata" "--target" "x86_64-unknown-linux-gnu" "--color" "always" "-"

That arg file contains

--crate-type=bin
--cfg=miri
--sysroot=/home/r/.cache/miri
-Ldependency=/home/r/src/rust/miri/test-cargo-miri/target/miri/x86_64-unknown-linux-gnu/debug/deps
-Ldependency=/home/r/src/rust/miri/test-cargo-miri/target/miri/debug/deps
--extern=byteorder_2=/home/r/src/rust/miri/test-cargo-miri/target/miri/x86_64-unknown-linux-gnu/debug/deps/libbyteorder-ee60f6ecbd530bd3.rmeta
--extern=byteorder=/home/r/src/rust/miri/test-cargo-miri/target/miri/x86_64-unknown-linux-gnu/debug/deps/libbyteorder-f178d57ea71fe0c8.rmeta
--extern=cargo_miri_test=/home/r/src/rust/miri/test-cargo-miri/target/miri/x86_64-unknown-linux-gnu/debug/deps/libcargo_miri_test-abd1c36147f6f274.rmeta
--extern=exported_symbol=/home/r/src/rust/miri/test-cargo-miri/target/miri/x86_64-unknown-linux-gnu/debug/deps/libexported_symbol-6d3a18b7c769098c.rmeta
--extern=issue_1567=/home/r/src/rust/miri/test-cargo-miri/target/miri/x86_64-unknown-linux-gnu/debug/deps/libissue_1567.rmeta
--extern=issue_1691=/home/r/src/rust/miri/test-cargo-miri/target/miri/x86_64-unknown-linux-gnu/debug/deps/libissue_1691-65d452f0d1689ec9.rmeta
--extern=issue_1705=/home/r/src/rust/miri/test-cargo-miri/target/miri/x86_64-unknown-linux-gnu/debug/deps/libissue_1705.rmeta
--extern=issue_1760=/home/r/src/rust/miri/test-cargo-miri/target/miri/debug/deps/libissue_1760-c4d1a96b539ee302.so
--extern=issue_rust_86261=/home/r/src/rust/miri/test-cargo-miri/target/miri/x86_64-unknown-linux-gnu/debug/deps/libissue_rust_86261-22dc9d7df84cf40e.rmeta
--extern=serde_derive=/home/r/src/rust/miri/test-cargo-miri/target/miri/debug/deps/libserde_derive-bc2328ee0791f66b.so
-Ccodegen-units=1
-Cembed-bitcode=no
-Zunstable-options

So... these arguments actually are all already patched properly as we are patching the --extern etc before invoking rustdoc.

The only actual issue is that Miri needs to run the rustc driver with a different default sysroot, and there's currently no way to do that other than patching command-line arguments. And annoyingly rustc bails when there are multiple --sysroot so we have to scan the arguments to see if some other sysroot is already set and of course that fails when there are argument files.

Ideally rustc_driver would just let us set a default sysroot so we don't need these command-line parsing hacks. (Well we still need them in cargo-miri but at least not in the miri driver. So in the future arg files may still cause more issues.)

[RalfJung]

Cc @GuillaumeGomez

It seems like rust-lang/rust#122993 may help though (hard to test since I haven't yet managed to run cargo-miri with doctests inside the rustc sysroot). But there could still be various other things going wrong in cargo-miri as it relies quite a bit on being able to scan the command-line arguments passed to rustc...

Does cargo itself ever use arg files?

[dtolnay]

For anyone looking for a workaround: use cargo miri test --all-targets for now, which skips doctests but includes unit tests, examples, benches, and integration tests in the library and all binaries as cargo miri test would.

[RalfJung]

Looking at what cargo-miri does here, even if rust-lang/rust#122993 resolves the --sysroot flag issue, there is still some command-line scanning going on for "the rustc invoked by rustdoc". So here are some flags where if those were passed inside an arg file, everything breaks:

• --target
• --crate-type=lib (I see you are passing the --crate-type=bin inside the arg file, that happens to work since bin is also the default when there is no flag -- Miri sees no flag and assumes the default)
• --test
• --emit
• -o, --out-dir
• --crate-name
• -C extra-filename

Will rustdoc ever pass any of these in an arg file?

Probably Miri should "just" implement arg file handling, but not sure when we'll get around to doing that.