A typo can lead to accidentally skipping all tests

[dpc]

I've discovered that our CI was not running quite a few of test suites we have. It was weird because when running locally they were being ran, but in the CI they were not.

After quite a lot of debugging I discovered that a typo can lead to strange behavior. The typo was that if the CARGO_PROFILE env var was set (to a custom ci profile in the CI), the scripts would pass -profile ci (instead of the desired --profile ci).

This leads to logs of tests being skipped:

> cargo nextest run --cargo-profile ci -profile ci --workspace --all-targets
    Finished ci [unoptimized + debuginfo] target(s) in 0.23s
    Starting 2 tests across 50 binaries (176 skipped)
        PASS [   0.004s] fedimint-mint-client tests::select_notes_uses_big_note_if_small_amounts_are_not_sufficient
        PASS [   0.011s] tbs tests::test_insufficient_shares
------------
     Summary [   0.012s] 2 tests run: 2 passed, 176 skipped

I still don't know how is this being parsed. I thought -p might be considered cargo's --package/-p, but we don't have a rofile package, so why wouldn't it fail?

[sunshowers]

Hmm, so cargo build -profile --workspace also works, even though rofile doesn't match any packages -- I think this should be detected and at least warned about, possibly eventually made a failure, in upstream Cargo.

I think the fact that we accept filter arguments over the CLI comes back to bite us (the ci gets interpreted as a filter). Perhaps that was a mistake: we should always have required filters to be passed in after --, and we should consider deprecating filters passed in before -- at some point.

[sunshowers]

I've proposed #1109 to deprecate and disable test name filters before --. You're not the first and probably not the last to run into this.

[dpc]

Thanks. All makes sense. I guess this issue can be closed as the new issues are more specific and cover everything.