Skip to content

Better control of test-execution order #6266

Open

Description

Cargo currently does not take too much care about what order tests are executed. cargo_test::compile_tests() places them in order of (PackageID, TargetKind, Name). We could do better than that by ordering test execution by

  1. If the test has failed before
  2. If the test has been executed at all
  3. (PackageID, TargetKind)
  4. The time it took the last time to execute this test, in descending order
  5. Name

This has some benefits:

  • Failing tests are executed first, without the need to stop cargo watch and filter the test we just broke.
  • Tests which have not been executed at all (due to fail-fast) are executed as soon as possible, so we get a least one reading
  • Slow tests are scheduled first, so their execution overlaps as long as possible, avoiding having just one or two slow tests be scheduled at the end and consuming just one cpu.

Notice that if the testsuite completes successfully, the ordering almost returns to current behaviour, with only latency-scheduling taking precedence over Name.

To do this, we'd need to persists some information to disk. We could do this by writing a "marker"-file somewhere in /target for every test (consider "target/testruns/test-hash(...)") where the files' ctime is the time the test were last started and the files' mtime is the time the tests were last completed successfully. We can reconstruct the above ordering from that.
Alternatively we serialize a single struct to disk.
Also notice that the ordering we derive from this information may be wrong or outdated (e.g. by changing test-bodies), yet this does not cause the testsuite to execute incorrectly as a whole and Is Only Wrong Once™

See also #10673

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    C-feature-requestCategory: proposal for a feature. Before PR, ping rust-lang/cargo if this is not `Feature accepted`Category: proposal for a feature. Before PR, ping rust-lang/cargo if this is not `Feature accepted`Command-testS-blocked-externalStatus: ❌ blocked on something out of the direct control of the Cargo project, e.g., upstream fixStatus: ❌ blocked on something out of the direct control of the Cargo project, e.g., upstream fix

    Type

    No type

    Projects

    • Status

      No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions