Stop rebuilding crates during testing

[brson]

We have unit tests inline with crates, and testing them involves recompiling the crates in test mode. This is bad for development times. We are currently doing this for core, std, rustc, rustdoc, and cargo.

My preference is just to always compile the crates with tests and use an external test runner to invoke them. It will add some compile time to non-testing builds, and some size to the binaries so maybe we can have a way to turn them off.

We could also move all tests to their own libraries, but it would preclude testing private functions.

[nikomatsakis]

Not critical for 0.6; removing milestone

[graydon]

Hm, tricky. coretest is 700k larger than core (5.2m vs. 4.5m) and stdtest is 4mb larger than std (9.0m vs. 5.0m). I think increasing our binary distribution by that much is pretty unappealing, and even if limited to maintainer-mode, it'd mean probably doubling cycle time for in-workspace development before you start running the unit tests.

[pnkfelix]

@graydon What do you think about this other point brson made: "We could also move all tests to their own libraries, but it would preclude testing private functions." ?

In particular, we could leave the private function tests in place, but move the bulk (well, hopefully the bulk) of the tests out into separately built libraries?

[graydon]

It's plausible, yeah. Arguably the public interface is the right level to be doing unit tests at anyways (private tests may be too brittle). One can also design stable "test interfaces" into libraries for cases where this wouldn't otherwise be so: design-for-test is a real thing.

How many private library tests do we have presently?

[huonw]

We now have make NO_REBUILD=1 which almost always works for the conventional edit-test cycle in a single crate IME (some people say that it doesn't work reliably for them so there may still be problems with it), but nothing beyond this.

[steveklabnik]

Triage: no updates. I'm not sure this is somthing that will ever get implemented.

[alexcrichton]

This is quite an old bug at this point, and I think that we've got enough options for this today. With Cargo we can have external test suites in tests/*.rs and otherwise upcoming incremental compilation I believe may render this issue moot anyway