Running doc tests are slow due to being always being recompiled

[Marwes]

Originally spawned by an issue to rust-skeptic which has the same problem.

Testing my combine library which has 44 doc tests and 36 normal tests takes ~26 seconds with ~22 of those being spent in doc tests. From what I can tell this is because cargo recompiles these tests every time cargo test is run in addition to each test being compiled individually.

I am thinking this is for a good reason since it makes sure two doc tests cannot conflict with each other but it would be nice if it could be improved.

[alexcrichton]

Yeah this is an unfortunate artifact where if Cargo sees a file change it doesn't actually know what changed in that file. To account for this Cargo takes the most conservative approach and assumes the library changed, so it recompiles the library. You may be able to use cargo test --doc to shave off compiling other unit tests, but otherwise, this probably won't get much of a boost until incremental compilation lands which is where the compiler should detect only a doc block changed and won't actually recompile anything (hopefully)

[Marwes]

Ok, guess its waiting for incremental compilation then at least. I suppose there is no way to tap into the incremental compilation framework once it is done for the skeptic case?

[alexcrichton]

Yeah I don't think the compiler will have any sort of API which Cargo can use to query whether a change was doc-related or not, but hopefully the compiler can very quickly realize it's not supposed to do anything!

[brson]

I proposed a solution for this in skeptic that I think works, or at least solves one of the problems in the cargo+rustdoc case. One of the problems is that each (libtest) test case compiles each (doc) test case every time the test is run. In skeptic's case it can be modified to cache those compiled test cases in cargo's output directory.

In rustdoc's case it's not as simple since rustdoc can't count on being run from cargo. But you could imagine a flag to rustdoc like --cache-dir that caches the compiled test cases.

[dwijnand]

Was this labeled blocked, waiting for the incremental compiler? If so, it shipped in Rust 1.24: https://github.com/rust-lang/rust/blob/master/RELEASES.md#version-1240-2018-02-15.

At which point, is this issue still valid?

[Marwes]

While the timing has vastly improved rustdoc still compiles each test on each cargo test --doc invocation (I can still see the linker getting started for instance).

For reference, combine which previously had 44 doc tests compiled in 22 seconds now has 135 doc tests and are compiled in 13 seconds.