Tracking issue for eRFC 2318, Custom test frameworks

[Centril]

This is tracking the #[bench] and #[test_case] attributes and the custom_test_frameworks lang item as well as the libtest crate. There is currently no plan/RFC for stabilizing them.

Old issue description.

This is a tracking issue for the eRFC "Custom test frameworks" (rust-lang/rfcs#2318).

Documentation:

• Implementation proposal: https://www.jrenner.net/rust/testing/2018/08/06/custom-test-framework-prop.html
• https://doc.rust-lang.org/nightly/unstable-book/language-features/custom-test-frameworks.html

Steps:

• Implement the RFC (cc @rust-lang/dev-tools @rust-lang/cargo -- can anyone write up mentoring instructions?)
• Adjust documentation (see instructions on forge)
• Stabilization PR (see instructions on forge)

Unresolved questions:

• Integration with doctests
• Standardizing the output
• Namespacing
• Runtime dependencies and flags
• Naming
• Bencher
• Cargo integration

Notes:

• Destabilization of #[bench]: [1.38] #[bench] accepted on stable but not beta #63798 (comment)

Implementation history:

• Initial implementation: Introduce Custom Test Frameworks #53410

[CAD97]

On stdout/err capture (after doing some research around the proof-of-concept #50457):

The current state is sub-optimal, to say the least. Both stdout and stderr exist as thread-local handles that only the stdlib has access to, which print family of macros use, as does panic.

Relevant libstd snippets

rust/src/libstd/io/stdio.rs

Lines 23 to 27 in 935a2f1

	thread_local! {
	static LOCAL_STDOUT: RefCell<Option<Box<Write + Send>>> = {
	RefCell::new(None)
	}
	}

rust/src/libstd/panicking.rs

Lines 38 to 42 in 935a2f1

	thread_local! {
	pub static LOCAL_STDERR: RefCell<Option<Box<Write + Send>>> = {
	RefCell::new(None)
	}
	}

rust/src/libstd/io/stdio.rs

Lines 614 to 711 in 935a2f1

	/// Resets the thread-local stderr handle to the specified writer
	///
	/// This will replace the current thread's stderr handle, returning the old
	/// handle. All future calls to `panic!` and friends will emit their output to
	/// this specified handle.
	///
	/// Note that this does not need to be called for all new threads; the default
	/// output handle is to the process's stderr stream.
	#[unstable(feature = "set_stdio",
	reason = "this function may disappear completely or be replaced \
	with a more general mechanism",
	issue = "0")]
	#[doc(hidden)]
	pub fn set_panic(sink: Option<Box<Write + Send>>) -> Option<Box<Write + Send>> {
	use panicking::LOCAL_STDERR;
	use mem;
	LOCAL_STDERR.with(move |slot| {
	mem::replace(&mut *slot.borrow_mut(), sink)
	}).and_then(|mut s| {
	let _ = s.flush();
	Some(s)
	})
	}
	/// Resets the thread-local stdout handle to the specified writer
	///
	/// This will replace the current thread's stdout handle, returning the old
	/// handle. All future calls to `print!` and friends will emit their output to
	/// this specified handle.
	///
	/// Note that this does not need to be called for all new threads; the default
	/// output handle is to the process's stdout stream.
	#[unstable(feature = "set_stdio",
	reason = "this function may disappear completely or be replaced \
	with a more general mechanism",
	issue = "0")]
	#[doc(hidden)]
	pub fn set_print(sink: Option<Box<Write + Send>>) -> Option<Box<Write + Send>> {
	use mem;
	LOCAL_STDOUT.with(move |slot| {
	mem::replace(&mut *slot.borrow_mut(), sink)
	}).and_then(|mut s| {
	let _ = s.flush();
	Some(s)
	})
	}
	/// Write `args` to output stream `local_s` if possible, `global_s`
	/// otherwise. `label` identifies the stream in a panic message.
	///
	/// This function is used to print error messages, so it takes extra
	/// care to avoid causing a panic when `local_stream` is unusable.
	/// For instance, if the TLS key for the local stream is
	/// already destroyed, or if the local stream is locked by another
	/// thread, it will just fall back to the global stream.
	///
	/// However, if the actual I/O causes an error, this function does panic.
	fn print_to<T>(
	args: fmt::Arguments,
	local_s: &'static LocalKey<RefCell<Option<Box<Write+Send>>>>,
	global_s: fn() -> T,
	label: &str,
	)
	where
	T: Write,
	{
	let result = local_s.try_with(|s| {
	if let Ok(mut borrowed) = s.try_borrow_mut() {
	if let Some(w) = borrowed.as_mut() {
	return w.write_fmt(args);
	}
	}
	global_s().write_fmt(args)
	}).unwrap_or_else(|_| {
	global_s().write_fmt(args)
	});
	if let Err(e) = result {
	panic!("failed printing to {}: {}", label, e);
	}
	}
	#[unstable(feature = "print_internals",
	reason = "implementation detail which may disappear or be replaced at any time",
	issue = "0")]
	#[doc(hidden)]
	pub fn _print(args: fmt::Arguments) {
	print_to(args, &LOCAL_STDOUT, stdout, "stdout");
	}
	#[unstable(feature = "print_internals",
	reason = "implementation detail which may disappear or be replaced at any time",
	issue = "0")]
	#[doc(hidden)]
	pub fn _eprint(args: fmt::Arguments) {
	use panicking::LOCAL_STDERR;
	print_to(args, &LOCAL_STDERR, stderr, "stderr");
	}

The existing stdout/err capture for the test harness works by setting the thread-local, which, if set, the print family of macros will use, otherwise falling back to io::stdout(). This setup is problematic because it's fairly trivial to escape this (#12309) and 99.9% of logging frameworks will escape accidentally (#40298).

The reason for this is that any solution that's generic over io::Write is going to use io::stdout() as the sink to hit stdout, and that doesn't go through the tread-local plumbing, so hits the true stdout directly.

The way I see it, there are three ways to resolve this quirk:

• Create a new stdlib io::Write sink that uses the thread-local plumbing that the print macros do today, and implement the print macros as writing to that sink. That's the approach I proved was possible with Reusable handle to thread-local replaceable stdout/err #50457.
• Create a new io::Write sink that uses (e)print! behind the scenes. This is possible entirely in an external crate:

(E)PrintWriter - feel free to use these, I am the author and release them under MIT/Unlicense

/// IO sink that uses the `print` macro, for test stdout capture.
pub struct PrintWriter;
impl std::io::Write for PrintWriter {
    fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
        print!("{}", String::from_utf8_lossy(buf));
        Ok(buf.len())
    }
    fn flush(&mut self) -> std::io::Result<()> { std::io::stdout().flush() }
}

/// IO sink that uses the `eprint` macro, for test stdout capture.
pub struct EPrintWriter;
impl std::io::Write for EPrintWriter {
    fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
        eprint!("{}", String::from_utf8_lossy(buf));
        Ok(buf.len())
    }
    fn flush(&mut self) -> std::io::Result<()> { std::io::stdout().flush() }
}

Unfortunately, this suffers from a large number of drawbacks. Because we only have access to the print macro, we have to interpret whatever sequence of bytes the sink gets as a UTF-8 string, even though that's potentially not true, etc. etc. not zero-cost when it's not, and so on. It's a decent solution for hooking up logging in tests, but is very much not a general solution that could be used outside tests.

That takes us to the third option, which is almost certainly the best, but also the most complicated:

• Actually change stdout/err

It sounds simple, but it's a lot simpler said than done with the current test harness setup. io::Stderr boils down to a handle to a write stream available to the process, and is a process-local resource -- there's only one stdout per process. The test harness currently runs tests parallel on different threads within the same process, thus the current thread-local replacement.

The other issue is that the test harness doesn't just want to capture stdout of the tests, it wants to capture stdout of the tests separately for each test, so it can print just the stdout of the failing test(s) and not have the other tests' output interspersed with it.

The only solution I see at this time that maintains current behavior of separate captures per test run is to run them all in their own process, and capture stdout that way. That way each process can pipe its stdout/err to the test runner, which can then synchronize them and decide which ones to output to the true stdout/err if desired.

A sketch of the plumbing required to make an out-of-tree libtest that works with this behavior:

• #[test] fn are all collected into a list
• A fn main is generated that takes an argument of one of the paths, and runs that test
• (The existing fn main is hidden)
• As normal, successful execution is success, and a panic is a failure exit code
• Our runner calls the generated binary with each of the test paths that match any filtering done as separate processes, captures their stdout/err and reports them on failure

Note that this is unfortunately stateful: we need to fold over all #[test] fn before we can emit the proper main. I'm not knowledgeable enough around compiler internals to be much help implementing the machinery right now (though I'd love to learn my way around and help out!), but give me an API to build against and I'll definitely help with any test frameworks built against the plumbing the eRFC has us set to experiment with.

[CAD97]

TL;DR:

To have sane capture behavior for stdout and stderr for print-family macros, panic, and io::{stdout, stderr}, running each test in its own (child) process is probably required. I'd love to help out wherever I can, but lack confidence.

[retep998]

The only solution I see at this time that maintains current behavior of separate captures per test run is to run them all in their own process, and capture stdout that way.

This would be significantly slower on certain platforms where process creation is slow, notably Windows. Especially when the tests are all very quick this will cause severe regressions in total time.

[CAD97]

Another possibility for process separation of tests is to spawn a process for each unit of parallelism, but each process is reused for multiple tests.

Of course, we could also just promote the thread-local replaceable stdout/err to io::Stdout/Err, but that would be suboptimal itself.

The issue is that stdout is, as far as I can tell, a process-local resource. To capture io::Stdout we either need our own stdout replacement machinery on top or to separate processes.

[SoniEx2]

if I want to argue about this do I argue about it here?

so like I made hexchat-plugin crate. the problem is you can't test hexchat side without hexchat. so I'd need some sort of #[hexchat_test].

but I still want the users to be able to use plain old vanilla rust #[test]s. I shouldn't have to reimplement/replace the whole testing system just to be able to produce some hexchat-based unit tests (in addition to normal tests).

(disclaimer: I don't fully understand the RFC, but it seems to require either replacing everything, or not having tests)

[jonhoo]

@SoniEx2 this RFC is about how we'd like to change the underlying mechanism that is used to arrange testing in Rust so that authors can write new test frameworks (e.g., quickcheck) that have access to the same mechanisms as existing built-in tests ( able to access private functions, can find all tagged functions, etc.). The current proposal is relatively wholesale (you change the test framework for a whole crate), but this is something that may be improved upon. We decided against more sophisticated designs in the initial implementation phase (this is just an "experimental" RFC) so that we don't bite off more than we can chew, but I think down the line we'd want more fine-grained test framework selection too.

As for your specific example, you don't actually need custom frameworks for that feature :) It seems like you want to only run a test if hexchat isn't running? To do that, you could either mark the test with #[allow_fail], or you could have the test just return if it fails to connect. If you wanted a more fine-grained approach, you could write a procedural macro that re-writes any function annotated with #[hexchat_test] into a #[test] that returns if hexchat isn't running.