Option for a non-zero exit status for a failed unknown test

[nphmuller]

I recently ran into the issue described here: #141
Basically, test2json couldn't parse a test result (because I used fmt.Printf() without a linebreak), which caused gotestsum to report the test as failed (unknown).

This seems perfectly fine behavior, but I'd really like the tests to fail if a test returns a failed (unknown) status, so we can improve the code. Currently however gotestsum still exists with exit code 0 if this occurs.

[dnephin]

Thanks for opening this issue! I've run into this problem a few times. gotestsum always uses the exit status of the go test command, which is why it current exits with 0. I think this approach has worked well because while test2json can sometimes struggle to parse things correctly, the exit status of go test is very reliable and will always correctly communicate if the tests passed or failed.

It seems strange to me to report the tests as failing when they haven't. gotestsum reporting failed (unknown) is definitely unfortunate and misleading, but I haven't found a good way to fix that yet. I've been hoping that test2json becomes more reliable and the problem will go away.

I'd like the keep the current behaviour by default, because I do think it is correct, but I understand your desire to be able to find these problems sooner (instead of waiting for some other test to fail and then notice the extra failure in the output). A command line flag to exit non-zero when there are some test failures is an option. It would conflict with the -rerun-fails flag, but I guess we could prevent them from being set together.

Another option might be to make the failed (unknown) output less misleading. There are a few different test2json bugs that can cause it to happen, but maybe we could communicate that a little better than "unknown".