Add testing supports tests those need multiple compile commands

[lht]

For example, I need to test the re-export issue (#1115). Steps are:

• compile a crate as library, which re-exports a function.
• call that re-exported function from another crate.
• verify compile-pass or run-pass, etc.

This requires executing rustc twice, and we don't have this kind of support in the test framework now.

I believe there are more places where combining rustc and shell commands is needed. Like verifying metadata dumped by "rustc --ls". Check rustc generated filenames.

And add missing tests when the infrastructure is ready.

[brson]

The main thing we need this for is to compile a crate, then run a test that depends on that crate.

[graydon]

I'm not sure I understand what this bug is about. Can you clarify?

[lht]

I updated the description. Please feel free to reword it.

[lht]

Thought about this more... How about put tests in a single file that could be preprocessed by test runner.

When run the test, the test runner will:

• Create a directory for holding artifacts of this test.
• Scan and split this input delimited by "file:" directive, and save them to the above directory or feed to rustc directly.
• Compile to libraries or executable according to the "compile-flags" directive
• If there is a "run:" directive, execute the command line after compilation and verify the exit code

Below are two examples:

// Test rustc correctly handle input file with relative path
// and outputs properly named library.

// file: foo.rs
// compile-cmd:--lib ./foo.rs
fn main() { }

// run:ls libfoo-8e702f2a2bfac15b-0.0.so

// test re-export works across several crates

// file:foo.rs
// compile-flags:--lib
use std;
mod o {
  fn test() { std::io::println("foo::test"); }
}

// file:bar.rs
// compile-flags:--lib
use foo;
import foo::o::test;
export test;

// file:baz.rs
// compile-flags:--lib
use bar;
import tz = bar::test;
export tz;

// file:main.rs
// compile-flags:-L .
use baz;
fn main() {
  baz::tz();
}

// run:./main

[brson]

There are two issues being solved here: 1 is expressing dependencies between test crates, 2 is running arbitrary commands to test various properties.

Let's avoid adding a feature that instructs the test runner to execute arbitrary commands, because it's too open-ended and could make things harder to maintain later.

In your example above where you want to test that rustc names the output correctly, I would suggest that the best way to do this would be to write unit tests in rustc itself. That would verify that the functions that are responsible for naming output produce the correct result. It's true that it wouldn't strictly test that the actual file showed up as expected, but you can still get a lot of guarantees out of fine-grained unit tests.

Frankly, I'm not crazy about having the test runner create a workspace for every test and split them up like that. Not sure why, but it seems like too many moving parts.

For expressing dependencies between crates we could do something like the following:

1. We create a way to use rustc to query the attributes of source code.
2. We convert all the current comment-directives to attributes.
3. We add a #[test_dep] attribute that specifies the name of a different test, which compiletest then uses to establish a build order.
4. The existing #[crate_type] attribute would direct rustc to build the dependencies as libraries.

This replaces our little but growing comment-language with an existing mechanism.

I'm still not sure this is a good idea either for a few reasons:

1. The crates which are dependencies would then need to be marked in such a way that compiletest knows they aren't themselves tests. This is potentially fine, since we've had other cases like this - there's at least one file that is marked xfail just so compiletest doesn't run it.
2. We then have attributes in most every test. This could interfere with tests that are actually trying to test attributes
3. Attributes depend on the parser, so compile-fail tests that test parser failure would make it difficult to read the attributes.

After writing all those negatives I am less fond of this idea.

[lht]

I agree with the comment of running arbitrary commands. I prefer testing the functionality as unit tests.

Make a temporary work space for each test creates a "namespace" for crate libraries. So we can have lots of libfoo without wasting time finding new name or solving name collisions. A work space can also be created by putting many source files in one folder, then we need to invent another format for specifying the build order. Writing them all down in one file implies the build order. And it's easier to read when all code are in one file, presumably they are all short enough.

I actually like the "comment directives", it has limitations but it's also more intuitive and self-contained. The compiler is just a hundred lines of code? Attributes may be well used for unit tests, but I feel comment directive serves better when testing the compiler as a black box.

I am thinking this all as one single test. Building a dependent relationship makes things more complex, and maybe unnecessary.

[nikomatsakis]

I really need something like this for CCI. @brson and I discussed it and we were thinking of doing the following, which seems like "the simplest thing that could possibly work":

• Add a "//build --lib foo/bar.rs" directive. This will cause rustc to be invoked on the given target with the given options.
• There will be a sibling directory of run-pass, compile-fail, etc called "aux". When you want to write a test that uses multiple modules, you put the main module in run-pass etc as usual, and the other module(s) in aux. The other module(s) must be buildable.
• The main module can then do //build --lib aux/foo and use foo; as normal. The test runner will add -L aux to the command-line automatically.

[nikomatsakis]

Sorry, that's not quite right: probably the aux directory will be a sub-directory of run-pass and friends? Well, anyway, something like that. I plan to implement this soon.

[nikomatsakis]

Also, //build will probably just take a filename: //build aux/foo.rs. The options will be automatic.

[nikomatsakis]

Sorry for the many small comments... one last thought, probably just //build foo.rs, and let the aux directory be automatically supplied as an argument to the test runner.

[catamorphism]

Is this done? We have the aux-build attribute now, and I can't tell whether there's more beyond that that we want to implement.

[nikomatsakis]

I think it's done enough. Going to close (feel free to re-open).