Assert.Fail() does not publish failed test #726

MurrayLong · 2020-02-17T06:52:00Z

I have a single test failing with Assert.Fail();
But no failures are listed in the trx output, as below.
Passing a message argument to Assert.Fail() fixes the problem.

test.cs:

    [TestFixture]
    public class ExampleTest
    {
        [Test]
        public void This_test_should_fail()
        {
            Assert.Fail();
        }

        [Test]
        public void This_test_should_pass()
        {
            Assert.Pass();
        }
    }

>dotnet test --logger trx --results-directory .
...
Test Run Failed.
Total tests: 2
     Passed: 1
     Failed: 1

>cat *.trx
...
 </TestLists>
  <ResultSummary outcome="Completed">
    <Counters total="1" executed="1" passed="1" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />

The text was updated successfully, but these errors were encountered:

rprouse · 2020-02-28T02:49:34Z

We'll need to repro this in the NUnit Console and determine if it is an NUnit issue, an adapter issue or a dotnet test issue.

rprouse · 2020-02-28T03:24:59Z

First pass, running this as a .NET 4.7.2 test in Visual Studio works fine, so it isn't NUnit or the NUnit Adapter.

I'm also getting the same result with a .NET Core test assembly in Visual Studio.

dotnet test is also working;

dotnet test
...
Test Run Failed.
Total tests: 2
     Passed: 1
     Failed: 1
 Total time: 1.1454 Seconds

and I am getting the same results using the TRX logger,

dotnet test --logger trx --results-directory .
...
Test Run Failed.
Total tests: 2
     Passed: 1
     Failed: 1
 Total time: 1.1454 Seconds

Clearly something is different in your setup. I am using .NET Core 3.1. What about you?

How about the latest NUnit Adapter and Microsoft.NET.Test.Sdk, version 16.5.0?

MurrayLong · 2020-02-28T11:10:43Z

Microsoft.NET.Test.Sdk: 16.5.0
NUnit Adapter 3.16.1.0
NUnit: 3.12.0
TargetFramework: netcoreapp3.1

Did you check the contents of the .trx itself?

I see:
<Counters total="1" executed="1" passed="1" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />

rprouse · 2020-02-29T20:45:06Z

@MurrayLong, you are correct, I am seeing the same in the TRX file. This may be an NUnit Adapter issue, or it could be a VSTest issue. I will be moving this issue to the adapter.

This is also being tracked in the VSTest repo at microsoft/vstest#2319

rprouse · 2020-02-29T20:48:38Z

@OsirisTerje I've moved this here from NUnit as it is either an adapter issue or a VSTest issue. In the issue linked above in VSTest, they say that it works with 3.13 of the adapter.

@MurrayLong mentioning you so that you have a quick link to the new location of your issue.

nohwnd · 2020-03-11T11:17:38Z

@MurrayLong @rprouse I am not able to repro on neither 3.1.100 (TP 16.3) nor 3.1.200 (TP 16.5), please clarify which version of dotnet you are using by running dotnet --version Or ideally pin your version via global.json (dotnet new globaljson --sdk-version <your version>, remove bin and obj and zip and send the whole repro project by dropping it here. :) this is mine:
nunit1.zip

# file global.json
{
  "sdk": {
    "version": "3.1.100"
  }
}

<!-- file nunit1.csproj -->
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>

    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="nunit" Version="3.12.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.16.1.0" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0"/>
  </ItemGroup>

</Project>

// file UnitTest1.cs
using NUnit.Framework;

namespace nunit1
{
    [TestFixture]
    public class ExampleTest
    {
        [Test]
        public void This_test_should_fail()
        {
            Assert.Fail();
        }

        [Test]
        public void This_test_should_pass()
        {
            Assert.Pass();
        }
    }
}

C:\Projects\temp\nunit1> dotnet --version
3.1.100
C:\Projects\temp\nunit1> dotnet test --logger trx --results-directory .
Test run for C:\Projects\temp\nunit1\bin\Debug\netcoreapp3.1\nunit1.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
  X This_test_should_fail [134ms]
  Stack Trace:
     at nunit1.ExampleTest.This_test_should_fail() in C:\Projects\temp\nunit1\UnitTest1.cs:line 11

Test run in progress.Results File: C:\Projects\temp\nunit1\jajares_JAJARES-SURBOOK_2020-03-11_11_59_11.trx

Test Run Failed.
Total tests: 2
     Passed: 1
     Failed: 1
 Total time: 3,3069 Seconds
C:\Projects\temp\nunit1> type .\jajares_JAJARES-SURBOOK_2020-03-11_11_59_11.trx
<?xml version="1.0" encoding="utf-8"?>
<TestRun id="3125c522-4421-47bb-89a8-63ec690ef2f2" name="jajares@JAJARES-SURBOOK 2020-03-11 11:59:11" runUser="EUROPE\jajares" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2020-03-11T11:59:11.3294890+01:00" queuing="2020-03-11T11:59:11.3294919+01:00" start="2020-03-11T11:59:07.9678498+01:00" finish="2020-03-11T11:59:11.5390733+01:00" />
  <TestSettings name="default" id="1095a22d-7f6f-4bfd-95a4-4874da013a8c">
    <Deployment runDeploymentRoot="jajares_JAJARES-SURBOOK_2020-03-11_11_59_11" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="91a27f80-dbec-4ec2-a562-0f6fac65ec55" testId="15a2ecc0-02b2-6a7f-d027-3074ab95741f" testName="This_test_should_pass" computerName="JAJARES-SURBOOK" duration="00:00:00.0018750" startTime="2020-03-11T11:59:10.0000000+01:00" endTime="2020-03-11T11:59:10.0000000+01:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="91a27f80-dbec-4ec2-a562-0f6fac65ec55" />
  </Results>
  <TestDefinitions>
    <UnitTest name="This_test_should_pass" storage="c:\projects\temp\nunit1\bin\debug\netcoreapp3.1\nunit1.dll" id="15a2ecc0-02b2-6a7f-d027-3074ab95741f">
      <Execution id="91a27f80-dbec-4ec2-a562-0f6fac65ec55" />
      <TestMethod codeBase="C:\Projects\temp\nunit1\bin\Debug\netcoreapp3.1\nunit1.dll" adapterTypeName="executor://nunit3testexecutor/" className="nunit1.ExampleTest" name="This_test_should_pass" />
    </UnitTest>
    <UnitTest name="This_test_should_fail" storage="c:\projects\temp\nunit1\bin\debug\netcoreapp3.1\nunit1.dll" id="2a2e2546-9038-cc65-c034-21f198205f15">
      <Execution id="084f5158-d057-48b9-9c6b-ba33ec28c92f" />
      <TestMethod codeBase="C:\Projects\temp\nunit1\bin\Debug\netcoreapp3.1\nunit1.dll" adapterTypeName="executor://nunit3testexecutor/" className="nunit1.ExampleTest" name="This_test_should_fail" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="15a2ecc0-02b2-6a7f-d027-3074ab95741f" executionId="91a27f80-dbec-4ec2-a562-0f6fac65ec55" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
  <TestLists>
    <TestList name="Results Not in a List" id="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Completed">
    <Counters total="1" executed="1" passed="1" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
    <Output>
      <StdOut>NUnit Adapter 3.16.1.0: Test execution started&#xD;
Running all tests in C:\Projects\temp\nunit1\bin\Debug\netcoreapp3.1\nunit1.dll&#xD;
   NUnit3TestExecutor converted 2 of 2 NUnit test cases&#xD;
NUnit Adapter 3.16.1.0: Test execution complete&#xD;
</StdOut>
    </Output>
  </ResultSummary>
</TestRun>



C:\Projects\temp\nunit1> code .\global.json
C:\Projects\temp\nunit1> type .\global.json
{
  "sdk": {
    "version": "3.1.200"
  }
}
C:\Projects\temp\nunit1> dotnet --version
3.1.200
C:\Projects\temp\nunit1> del obj, bin -force -recurse
C:\Projects\temp\nunit1> dotnet test --logger trx --results-directory .
Test run for C:\Projects\temp\nunit1\bin\Debug\netcoreapp3.1\nunit1.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.5.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
  X This_test_should_fail [52ms]
  Stack Trace:
     at nunit1.ExampleTest.This_test_should_fail() in C:\Projects\temp\nunit1\UnitTest1.cs:line 11

Results File: C:\Projects\temp\nunit1\jajares_JAJARES-SURBOOK_2020-03-11_12_00_25.trx

Test Run Failed.
Total tests: 2
     Passed: 1
     Failed: 1
 Total time: 1,7543 Seconds
C:\Projects\temp\nunit1> type .\jajares_JAJARES-SURBOOK_2020-03-11_12_00_25.trx
<?xml version="1.0" encoding="utf-8"?>
<TestRun id="76e63fba-2939-449f-88f8-fb66a4a4ce67" name="jajares@JAJARES-SURBOOK 2020-03-11 12:00:25" runUser="EUROPE\jajares" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2020-03-11T12:00:25.8788965+01:00" queuing="2020-03-11T12:00:25.8788986+01:00" start="2020-03-11T12:00:24.0971514+01:00" finish="2020-03-11T12:00:25.9037876+01:00" />
  <TestSettings name="default" id="1f24449a-0fae-4645-8973-774520c24e50">
    <Deployment runDeploymentRoot="jajares_JAJARES-SURBOOK_2020-03-11_12_00_25" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="c2c21e2b-0f3b-400a-a918-b6976917c896" testId="15a2ecc0-02b2-6a7f-d027-3074ab95741f" testName="This_test_should_pass" computerName="JAJARES-SURBOOK" duration="00:00:00.0018490" startTime="2020-03-11T12:00:25.0000000+01:00" endTime="2020-03-11T12:00:25.0000000+01:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="c2c21e2b-0f3b-400a-a918-b6976917c896" />
  </Results>
  <TestDefinitions>
    <UnitTest name="This_test_should_pass" storage="c:\projects\temp\nunit1\bin\debug\netcoreapp3.1\nunit1.dll" id="15a2ecc0-02b2-6a7f-d027-3074ab95741f">
      <Execution id="c2c21e2b-0f3b-400a-a918-b6976917c896" />
      <TestMethod codeBase="C:\Projects\temp\nunit1\bin\Debug\netcoreapp3.1\nunit1.dll" adapterTypeName="executor://nunit3testexecutor/" className="nunit1.ExampleTest" name="This_test_should_pass" />
    </UnitTest>
    <UnitTest name="This_test_should_fail" storage="c:\projects\temp\nunit1\bin\debug\netcoreapp3.1\nunit1.dll" id="2a2e2546-9038-cc65-c034-21f198205f15">
      <Execution id="f5f0707c-fe49-4b9a-8624-da22044c0b86" />
      <TestMethod codeBase="C:\Projects\temp\nunit1\bin\Debug\netcoreapp3.1\nunit1.dll" adapterTypeName="executor://nunit3testexecutor/" className="nunit1.ExampleTest" name="This_test_should_fail" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="15a2ecc0-02b2-6a7f-d027-3074ab95741f" executionId="c2c21e2b-0f3b-400a-a918-b6976917c896" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
  </TestEntries>
  <TestLists>
    <TestList name="Results Not in a List" id="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Completed">
    <Counters total="1" executed="1" passed="1" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
    <Output>
      <StdOut>NUnit Adapter 3.16.1.0: Test execution started&#xD;
Running all tests in C:\Projects\temp\nunit1\bin\Debug\netcoreapp3.1\nunit1.dll&#xD;
   NUnit3TestExecutor converted 2 of 2 NUnit test cases&#xD;
NUnit Adapter 3.16.1.0: Test execution complete&#xD;
</StdOut>
    </Output>
  </ResultSummary>
</TestRun>

MurrayLong · 2020-03-11T14:55:44Z

@nohwnd You have reproduced.

compare trx:
<Counters total="1" executed="1" passed="1" failed="0" ...

to dotnet test output:

Total tests: 2
     Passed: 1
     Failed: 1

nohwnd · 2020-03-19T16:10:57Z

Was an error on our side. Should be fixed here.

microsoft/vstest#2319

OsirisTerje · 2020-03-19T16:58:03Z

@nohwnd Thanks!
You also wrote "This is a fix in the VSTest console it will be shipped with VS 16.6, possibly preview 2."

nohwnd · 2020-03-19T18:04:02Z

I did, is there a question? Or are you just copying the info so it's visible here as well? :)

OsirisTerje · 2020-03-19T19:14:28Z

Just copied it here for visibility :-)

rprouse transferred this issue from nunit/nunit Feb 29, 2020

rprouse mentioned this issue Feb 29, 2020

NUnit tests failure not reported in test results file microsoft/vstest#2319

Closed

OsirisTerje added the is:bug label Mar 11, 2020

OsirisTerje closed this as completed Mar 19, 2020

OsirisTerje added the closed:SomebodyElsesProblem Closed because there is nothing we can do, as this is caused by an external component label Mar 19, 2020

rprouse mentioned this issue Apr 3, 2020

TRX report is missing test result for a test that timed out #731

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Assert.Fail() does not publish failed test #726

Assert.Fail() does not publish failed test #726

MurrayLong commented Feb 17, 2020

rprouse commented Feb 28, 2020

rprouse commented Feb 28, 2020

MurrayLong commented Feb 28, 2020

rprouse commented Feb 29, 2020

rprouse commented Feb 29, 2020

nohwnd commented Mar 11, 2020

MurrayLong commented Mar 11, 2020

nohwnd commented Mar 19, 2020

OsirisTerje commented Mar 19, 2020

nohwnd commented Mar 19, 2020

OsirisTerje commented Mar 19, 2020

Assert.Fail() does not publish failed test #726

Assert.Fail() does not publish failed test #726

Comments

MurrayLong commented Feb 17, 2020

rprouse commented Feb 28, 2020

rprouse commented Feb 28, 2020

MurrayLong commented Feb 28, 2020

rprouse commented Feb 29, 2020

rprouse commented Feb 29, 2020

nohwnd commented Mar 11, 2020

MurrayLong commented Mar 11, 2020

nohwnd commented Mar 19, 2020

OsirisTerje commented Mar 19, 2020

nohwnd commented Mar 19, 2020

OsirisTerje commented Mar 19, 2020