dotnet-test prints error stacktrace outside context

[codito]

Description

dotnet test should print the stack trace for a failed test along with the Failed test. Currently it is printed after all the test results show up. This may confuse user.

Steps to reproduce

• Create a test project with few failing tests
• Run dotnet test

Expected behavior

Starting test execution, please wait...
Passed   SampleUnitTestProject.UnitTest1.PassingTest
Failed   SampleUnitTestProject.UnitTest1.FailingTest
Error Message:
 Assert.AreEqual failed. Expected:<2>. Actual:<3>.
Stack Trace:
   at SampleUnitTestProject.UnitTest1.FailingTest() in /home/codito/tmp/samples/mstesttest/Tests.cs:line 29
Skipped  SampleUnitTestProject.UnitTest1.SkippingTest
Passed   SampleUnitTestProject.UnitTest1.TestWithTestCategory
Passed   SampleUnitTestProject.UnitTest1.TestWithPriority
Passed   SampleUnitTestProject.UnitTest1.TestWithProperties
Failed   SampleUnitTestProject.UnitTest1.FailingTestWithTraits
Error Message:
 Assert.Fail failed.
Stack Trace:
   at SampleUnitTestProject.UnitTest1.FailingTestWithTraits() in /home/codito/tmp/samples/mstesttest/Tests.cs:line 70
Test Run Failed.

Actual behavior

Starting test execution, please wait...
Passed   SampleUnitTestProject.UnitTest1.PassingTest
Failed   SampleUnitTestProject.UnitTest1.FailingTest

Skipped  SampleUnitTestProject.UnitTest1.SkippingTest
Passed   SampleUnitTestProject.UnitTest1.TestWithTestCategory
Passed   SampleUnitTestProject.UnitTest1.TestWithPriority
Passed   SampleUnitTestProject.UnitTest1.TestWithProperties
Error Message:
 Assert.AreEqual failed. Expected:<2>. Actual:<3>.
Stack Trace:
   at SampleUnitTestProject.UnitTest1.FailingTest() in /home/codito/tmp/samples/mstesttest/Tests.cs:line 29
Failed   SampleUnitTestProject.UnitTest1.FailingTestWithTraits

Error Message:
 Assert.Fail failed.
Stack Trace:
   at SampleUnitTestProject.UnitTest1.FailingTestWithTraits() in /home/codito/tmp/samples/mstesttest/Tests.cs:line 70
Test Run Failed.

Total tests: 7. Passed: 4. Failed: 2. Skipped: 1.
Test execution time: 11.9409 Seconds

Environment

vstest repo at 9da5eb8

[codito]

Triage: combine the Failed text and error message into stderr.

[codito]

(some more thoughts as I read more about this)

As per posix standard, stderr should be used for writing diagnostic/error messages. IMHO test failures don't fall into diagnostic category, they are part of the core functionality that runner provides (to run and report test results). It is expected some tests will pass, some will fail.

Secondly, putting failed tests on stdout allows other tools to process it from a pipe.

However, an adapter could fail to discover tests because it couldn't load an assembly; or testhost may not start. This is unexpected and should go into error. Similarly if the test assembly doesn't exist, message should be written into error stream. E.g. here's how cat command behaves for a non-existent file:

> cat /tmp/foo 1> /dev/null
cat: /tmp/foo: No such file or directory

> cat /tmp/foo 2> /dev/null
# no output, stderr is redirected