Support discoverability of test targets accepted by `cargo test --test`

[rrnewton]

Problem

Context: I have many test targets and want to narrow down the scope of cargo test while in the development loop. Regular test filtering still spends a bunch of time running targets that say ok. 0 passed.

This interaction is frustrating:

$ cargo test --test foobar
error: no test target named `foobar`

Ok, great, but please tell me what the set of test targets is or how to find it. In this case, I want to run an integration test corresponding to a file tests/foobar.rs, but not how to express that in the right format for --test. Let's check the help:

$ cargo test --help
...
        --test <NAME>...                             Test only the specified test target

That doesn't shine any more light on it. What is the set from which NAME is drawn? What's the "type", if you will?

Next I looked for something like --list-targets but didn't find it.

Finally, I look through the output of cargo test, hoping that it essentially serves the function of listing test targets. But the printouts during a test run look like this:

     Running target/debug/deps/foobar-ed66a800a301dfb1

And that's clearly not the string I'm supposed to provide to --test!

Solution

Here are three possible solutions:

1. Provide a cargo test --list and mention it from the cargo test --test error message.
2. Print out the list of literal target strings as part of the cargo test --test error message. This can be truncated if it's excessively long.
3. Print precise target strings in a clear way while running tests.

Related

It seems #8396 is the only other issue that mentions this error message.
There was also this very old #812 discussing the documentation for cargo test vs cargo test --test.

[dzfranklin]

This is not quite what you want @rrnewton , but the default test runner libtest has a way to list tests. You pass options to the test runner by adding them after --, so cargo test -- --list does nearly what you want (you can also use all the normal ways of filtering and see a list of what was filtered in).

[epage]

cargo test --test will print out the list of tests but this isn't mentioned by cargo test -h or cargo help test.

So I can see

• Having cargo test --test <name> provide suggestions like clap does
• Document cargo test --test

The downside to cargo test --list is that we specifically want the list for cargo test --test but a --list flag doesn't make the clear compared to any of the other types of tests that can be run.

[sanmai-NL]

@danielzfranklin In a workspace, at least, that only lists the source files that contain tests, not the test function names.

[sanmai-NL]

cargo-nextest supports this already:

cargo nextest list

[sanmai-NL]

@epage That invocation doesn't work in my case (cargo 1.66.0 (d65d197ad 2022-11-15) on Linux inside a workspace).

sanmai@server4:~/workspace$ cargo test --no-run --test
error: "--test" takes one argument.
No tests available.

sanmai@server4:~/workspace$ cargo test --test
error: "--test" takes one argument.
No tests available.

[dzfranklin]

@danielzfranklin In a workspace, at least, that only lists the source files that contain test, not the test function names.

To clarify @sanmai-NL I have no expertise here I'm just someone who needed to programmatically discover test functions for a project.

You can see the code in libtest that outputs the function names here.

It works in my version. cargo 1.70.0-nightly (15d090969 2023-03-21). I just ran cargo test -- --list in a workspace and got a listing of test function names (I copied the stdout into a gist).

It should work in your version too. I wrote code to parse it two years ago for a project, though I haven't used it since.

I happened to see a recent commit while I was looking at the source entitled Implementing "<test_binary> --list --format json" which from a cursory look seems to be exactly what you want.

[epage]

@sanmai-NL

That invocation doesn't work in my case (cargo 1.66.0 (d65d197 2022-11-15) on Linux inside a workspace).

Not sure what to say but it works for me in a workspace (virtual or not) or out of it

$ cargo -V
cargo 1.68.0 (115f34552 2023-02-26)
$ cargo test --test
error: "--test" takes one argument.
Available tests:
    builder
    derive
    derive_ui
    examples
    macros
    ui
$ cargo +1.64.0 test --test
error: "--test" takes one argument.
Available tests:
    builder
    derive
    derive_ui
    examples
    macros
    ui
$ cargo +1.60.0 test --test  # Note: I had to switch to an older version with an appropriate MSRV
error: "--test" takes one argument.
Available tests:
    builder
    derive
    derive_ui
    examples
    macros
    yaml

[epage]

@sanmai-NL I just noticed your error said No tests available.. Sounds like this is something related to your project. You either don't have any test binaries (only --lib or --doc) or there is some enumeration bug for your project and a minimal reproduction case would be needed

[sanmai-NL]

The workspace isn't mine. But say it only has lib tests, then this proves that command isn't generic for test function discovery. The nextest command is (for lib tests at least).

[epage]

This issue is about discovery of test targets usable with cargo test --test which was the context for my answer. If you are looking at enumerating all test binaries to enumerate all test functions, that is different and my answer will not help with that.

[sanmai-NL]

@epage I wasn't fully aware of the distinction in libtest of these 'integration tests' and other types of tests. Just was aware of doctests, benches and functions with the test attribute. (Quoted because defining automated testing semantics through structure is IMO quite ... restricting.)

@rrnewton Would you mind clarifying the title to refer to integration tests rather than the current more technical description? Helps understandability and findability of the issue.

So, I think the more general functional requirement would be something like: as a developer or test engineer in Rust, want a CLI report in terms of a machine readable list of all discovered tests by their full item paths. Here, test meaning every function that would be executed when cargo test is run on a crate or workspace given the libtest test harness.

If we want to broaden the issue, then suggest to change the title as well.