Provide an option to `cargo check` everything except benchmarks

[RalfJung]

IDE plugins like RA like to invoke cargo check with --all-targets to ensure that tests and examples are checked as well, not just the main crate sources. However, this often fails on stable since it will also check benchmarks, which essentially always need nightly features (Cc crossbeam-rs/crossbeam#890).

So I thought I could work around this by telling RA to run "cargo" "check" "--lib" "--bins" "--tests" "--examples" "--workspace" "--message-format=json" instead -- however, this command sadly will fail when there is no lib target. Is there a way to say "please check everything except for the benchmarks"?

The documentation claims that --all-targets is equivalent to specifying --lib --bins --tests --benches --examples, but that is wrong: --lib --bins --tests --benches --examples will fail when there is no lib target, but --all-targets will still work.

[RalfJung]

E.g. there could be a --libs flag that selects all the libs targets that exist -- it's going to be either 0 or 1, but that still leaves two options, and cargo currently doesn't have a uniform way to deal with this situation.

I'd rather not have to manually configure RA for each and every package and then remember to change the configuration when a libs crate is added to a package.

[weihanglo]

Sorry for the subtlety between these flags. --tests implies lib target is included as well, so I believe cargo check --tests is effectively the same as cargo check --lib --tests except checking the existence of lib target. Does cargo check --bins --tests --examples work for you?

[RalfJung]

What if my Cargo.toml contains

[lib]
test = false

will cargo check --tests then still also check the lib? And is that a stable guarantee I can rely on? The documentation doesn't say anything like this (as far as I can see).

[RalfJung]

FWIW for now I am simply sticking to nightly rustc for crates where cargo check --all-targets does not work on stable. But at least the documentation issue in --all-targets should still be tracked.

[weihanglo]

What if my Cargo.toml contains

[lib]
test = false

will cargo check --tests then still also check the lib?

No. It won't. See https://doc.rust-lang.org/stable/cargo/commands/cargo-test.html#target-selection (or you can check the doc with the old new friend cargo help check 😆). cargo check --tests will skip those flagged as test = false.

And is that a stable guarantee I can rely on? The documentation doesn't say anything like this (as far as I can see).

I believe it's fairly stable. At least it's a documented behaviour and changes against that should be under meticulous reviews.

[RalfJung]

No. It won't.

That doesn't achieve my goal then, of checking everything except for the benchmarks.

[epage]

Personally, I would lean towards the following route until benches are stable

I would accept a patch to use criterion in benchmarks, but there is no motivation on our part to address this since this situation is so common.

[weihanglo]

Oh now I see. You want to run all Cargo targets including those disabled by default. I would suggest the following command:

cargo check --test '*' --bin '*' --example '*'

And I totally agree using criterion is a viable choice.

[ehuss]

Can you say more about why RA wants to check all targets? Like, in what situation does it do that? Would it be possible for it to run checks for a narrower subset?

[RalfJung]

Well usually I'd like my IDE to tell me when I do some change that breaks examples/tests/benchmarks. So it makes a lot of sense to check all targets I think?

In Miri I used to accidentally break benchmarks fairly regularly and only got caught on CI when I created a PR, which was rather annoying.

[RalfJung]

Note that ./x.py check library in rustc also checks tests and all the other targets, for the same reason.

[weihanglo]

will cargo check --tests then still also check the lib?

Oh wait. It should check the lib, but won't test it. Since test targets depends on the lib target, it will always get the lib target checked. Sorry for misreading your requirements.

[RalfJung]

cargo check --test '*' --bin '*' --example '*'

But that will still skip lib crates if they have no tests, I would assume?
In the extreme case, this crate might have only a lib crate and no tests/bins/examples at all. Then I would assume cargo check --test '*' --bin '*' --example '*' does nothing?

I should probably just test this locally...
... yes my theory is confirmed:

$ cargo check --tests
warning: Target filter `tests` specified, but no targets matched. This is a no-op
    Finished dev [unoptimized + debuginfo] target(s) in 0.21s

also FWIW

$ cargo check --test '*' --bin '*' --example '*'
error: no bin target matches pattern `*`

So this won't work at all for lib-only crates.

[RalfJung]

You want to run all Cargo targets including those disabled by default.

Yes. I want to do exactly what cargo check --all-targets does except I want to exclude benchmarks. And I want it to be the same command for all crates so that I can just configure RA globally to use that command and have it work everywhere.