libtest filters should include the test crate's name in the string matched against the filter

[RalfJung]

Let's say I have a whole bunch of .rs files in my tests folder, like:

• floats/lib.rs
• time.rs
• thread.rs
• ...

Now I want to run just the float tests. Naturally I'll do something like cargo test -- float. Sadly that does not work since this runs the tests that have float as a substring in the name of the test (mod1::mod2::test_fn), but the name of the crate is not included.

This is particularly confusing in the rustc repo itself, where ./x test library/{core,std} -- float will run the float tests in libcore but not the one in libstd: libcore has a single large test crate with a module called floats, while std has a bunch of separate test crates, one of which is called floats.

I think libtest should be changed to put the crate name in the string used for filtering, thus avoiding this confusing and surprising behavior.

@rust-lang/libs-api I've been told the libtest CLI API is in your purview. Is there any chance the behavior of filters could be adjusted? This would lead to strictly more tests being included in any given filter, which at least won't silently reduce test coverage (except when combined with --skip) -- but it could theoretically break a setup where that then includes a test that does not actually work.

[tgross35]

I've also been bitten by this before, would be happy to see this change. Cc @rust-lang/testing-devex based on the Zulip discussion.

[epage]

rust-lang/libs-api I've been told the libtest CLI API is in your purview. Is there any chance the behavior of filters could be adjusted?

@rust-lang/testing-devex is coordinating on efforts related to the long term direction of libtest, under the direction of libs-api.

I think libtest should be changed to put the crate name in the string used for filtering, thus avoiding this confusing and surprising behavior.

Currently, this is left up to the caller. For a cargo workflow, this would be cargo test --test floats.

I think libtest should be changed to put the crate name in the string used for filtering, thus avoiding this confusing and surprising behavior.

This would be a breaking change, particularly when --exact is passed in. One way around that would be a compatibility shim could be added where --exact matches both with and without the crate name. We'd have to track it separate than the test's identifier to make sure we didn't just turn --exact into a suffix match.

We'd also need to consider how this should be represented in json reporting. That isn't stable but we're working towards it.

We should also consider how this should work with --list which is stable.

As we clarify what the minimum expectations are for custom test harnesses, I feel like we'd need to maintain both behaviors because there is too much community expectations around the current behavior.

[epage]

I've re-opened rust-lang/testing-devex-team#12

[weihanglo]

I wonder if it is possible to put this kind of breaking change in the next edition, though having two filter logic might be more confusing.