Need help understanding `cargo test --jobs`

[eminence]

When I run cargo test --help, I see this option:

Options:
    -j N, --jobs N               Number of parallel jobs, defaults to # of CPUs

At first, it looks like this controls how many test cases run in parallel, but that doesn't seem to be the case. Given 5 different tests of this form:

#[test]
fn foo1() {
    println!("Running foo1");
    sleep(Duration::from_secs(5));
    println!("Done with foo1");
}

I would expect that running with --jobs 1 would take about 25 seconds. But all 5 tests actually are run in parallel:

achin@bigbox ~/tmp/16/foo $ time cargo test --jobs 1
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running target/debug/deps/foo-0213a102e10dc10f

running 1 test
test tests::it_works ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured

     Running target/debug/deps/foo-865370f5d0ccd0fc

running 5 tests
test foo3 ... ok
test foo1 ... ok
test foo5 ... ok
test foo2 ... ok
test foo4 ... ok
                                                                                                                                                                                                                    test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured

   Doc-tests foo

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured


real    0m5.316s
user    0m0.155s
sys     0m0.053s

Can you help me understand this? What does --jobs actually do? How can I get these 5 test cases to run serially?

Thanks!

cargo 0.20.0-nightly (cf17c9f71 2017-05-09)
release: 0.20.0
commit-hash: cf17c9f7118f544ec304ed6f50d92b3759487123
commit-date: 2017-05-09

[Mark-Simulacrum]

So this is actually a case of misleading help, to some extent. cargo test --jobs 1 means that Cargo will only compile crates/tests one at a time. You probably want cargo test -- --jobs 1 which will pass the number of jobs to the produced test binary, which will then run sequentially.

[eminence]

Ahh! That particular thing didn't work, but this did:

cargo test -- --test-threads 1

So what does --jobs actually do? How can this be documented better?

[Mark-Simulacrum]

Ah, yeah, sorry about that. --jobs is the amount of individual rustc instances cargo will run at once. I'm not really sure there's much we can do here, though... do you have any thoughts on how we could change the description of --jobs to make it clear that it's not the number of test threads?

[eminence]

Rereading the output of cargo test --help, I see the following text which I didn't notice before:

The --jobs argument affects the building of the test executable but does
not affect how many jobs are used when running the tests.

So there is definitely an element of user-error here (and so you could legitimately close this issue on that basis alone).

What if the option summary table just said something like "Number of parallel builds, see below for details", and in text below it said something:

The --jobs argument affects the building of the test executable but does
not affect how many jobs are used when running the tests. The default value for
the --jobs argument is the number of CPUs. If you want to control the number of
simultaneous running test cases, pass the test-threads option to the test binaries:

cargo test -- --test-threads `

[Mark-Simulacrum]

That seems better. Could you refile onto rust-lang/cargo please?