Cargo book doesn't specify which tests run in parallel and which tests run in fresh environment

[safinaskar]

Hi. I aware of 3 types of tests in Rust: unit tests, integration tests and doctests. I would like to see in documentation for cargo test in Cargo book and in Rustdoc book the following info: whether this tests run in parallel and whether every test is run in fresh environment, i. e. in new process created specially for this test. Please note that "sequential execution" and "execution of each test in fresh environment" are different things. Consider unit tests run using cargo test -- --test-threads=1. As well as I understand they will be run sequentially, but they still be run in the same process, thus chdir and similar operations from one test will affect others. I want to see in cargo book and other docs answers to this questions:

• Do unit tests run in parallel?
• Does every unit test run in fresh environment?
• Do integration tests (different #[test] functions from same file in tests) run in parallel?
• Does every integration test (different #[test] functions from same file in tests) run in fresh environment?
• Do integration tests from different files in tests run in parallel?
• Does every integration test (from different files in tests) run in fresh environment?
• Do doctests run in parallel?
• Does every doctest run in fresh environment?

Currently I was able to easily find answers to first two questions. But docs are vague on others. Section https://doc.rust-lang.org/rustdoc/documentation-tests.html doesn't answer questions on doctests

(moved from rust-lang/book#3042 and slightly edited)

[weihanglo]

Firstly, in cargo each target compiles down to a crate, which usually refers to a compilation artifact such like a library or a executable.

For unit test, cargo invokes rustc to collect all #[test] items and link them to libtest for building final executable. This executable runs all your unit tests inside a target and by default in parallel unless you pass cargo test -- --test-threads 1. For instance, if you have one lib target and one bin. When you run cargo test, Cargo produces two unittest executables: One for lib and the other for bin. The two executables are run one by one, just like how cargo-nextest describes in its own book. They are two processes spawn by cargo, so they might inherit a part of environment from cargo. But all #[test] inside the same target are just run under the same process. You can consult the relevant code here for more info.

The aforementioned rule also applies on integration tests, all #[test] items inside on test target are compiled into on libtest executable. Each integration test produces a crate so that each is run in different process. This seems somehow inefficient and people sometimes combine all integration tests into one target to exploit libtest's parallelism. (cargo itself does that as well)

Things get interesting in doc tests. Cargo permits only one lib target for each package, and only compiles doc test for lib target. You might ask, does that implies a package gets exactly one executable for running doc tests? The answer is No. Doc tests current are controlled by rustdoc, which leverages library artifact such like rlib to compile each code block into an executable respectively. So, instead of one executable, you get multiple exetuables if you have multiple code blocks to test. These exectuables are located in a temporary directory by default, so you won't see them in the target dir. The relevant code lies in here. Interestingly, code blocks of doctest are run in parallel although they are different executables.

Here is a table of my summary:

	Unit tests	Integration tests	Doctests
Run in parallel?	✅ for #[test] ❌ for target	✅ for #[test] ❌ for target	✅ for each code block
Share environment?	One target per process	One target per process	One code block per process

My knowledge might be rusty. Correct me if you find anything incorrect.

[weihanglo]

Close as #10726 adds more sentences on this topic. Thanks for the report!

cargo/src/doc/man/cargo-test.md

Lines 30 to 54 in cce487e

	Tests are built with the `--test` option to `rustc` which creates a special
	executable by linking your code with libtest. The executable automatically
	runs all functions annotated with the `#[test]` attribute in multiple threads.
	`#[bench]` annotated functions will also be run with one iteration to verify
	that they are functional.
	If the package contains multiple test targets, each target compiles to a
	special executable as aforementioned, and then is run serially.
	The libtest harness may be disabled by setting `harness = false` in the target
	manifest settings, in which case your code will need to provide its own `main`
	function to handle running tests.
	### Documentation tests
	Documentation tests are also run by default, which is handled by `rustdoc`. It
	extracts code samples from documentation comments of the library target, and
	then executes them.
	Different from normal test targets, each code block compiles to a doctest
	executable on the fly with `rustc`. These executables run in parallel in
	separate processes. The compilation of a code block is in fact a part of test
	function controlled by libtest, so some options such as `--jobs` might not
	take effect. Note that this execution model of doctests is not guaranteed
	and may change in the future; beware of depending on it.