-
-
Notifications
You must be signed in to change notification settings - Fork 473
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The NUnit report is invalid when some tests contain ForEach/TestCase data #2143
Comments
I have not seen this impact code coverage upload too tools like CodeCov.io. 🤔 Do you have a xml file that is generated erroneous and what the line should actually be? 🤔 |
Sure. Here's the output from the code above. <?xml version="1.0" encoding="utf-8" standalone="no"?>
<test-results xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="nunit_schema_2.5.xsd" name="Pester" total="4" errors="0" failures="0" not-run="0" inconclusive="0" ignored="0" skipped="0" invalid="0" date="2022-03-10" time="14:48:34">
<environment clr-version="4.0.30319.42000" user-domain="NA" cwd="C:\Temp\Test Pester NUnit Problem" platform="Microsoft Windows 10 Enterprise|C:\WINDOWS|\Device\Harddisk0\Partition3" machine-name="Computer" nunit-version="2.5.8.0" os-version="10.0.19043" user="User" />
<culture-info current-culture="en-GB" current-uiculture="en-GB" />
<test-suite type="TestFixture" name="Pester" executed="True" result="Success" success="True" time="0.1851" asserts="0" description="Pester">
<results>
<test-suite type="TestFixture" name="C:\Temp\Test Pester NUnit Problem\Test NUnit.ps1" executed="True" result="Success" success="True" time="0.1851" asserts="0" description="C:\Temp\Test Pester NUnit Problem\Test NUnit.ps1">
<results>
<test-suite type="TestFixture" name="Demonstrate NUnit Problem" executed="True" result="Success" success="True" time="0.0903" asserts="0" description="Demonstrate NUnit Problem">
<results>
<test-suite type="ParameterizedTest" name="Demonstrate NUnit Problem.Should return <Result> when type is <Type>" executed="True" result="Success" success="True" time="0.064" asserts="0" description="Should return <Result> when type is <Type>">
<results>
<test-case description="Should return True when type is String" name="Demonstrate NUnit Problem.Should return True when type is String" time="0.0362" asserts="0" success="True" result="Success" executed="True" />
<test-case description="Should return False when type is Int" name="Demonstrate NUnit Problem.Should return False when type is Int" time="0.0132" asserts="0" success="True" result="Success" executed="True" />
<test-case description="Should return False when type is Bool" name="Demonstrate NUnit Problem.Should return False when type is Bool" time="0.0146" asserts="0" success="True" result="Success" executed="True" />
</results>
</test-suite>
<test-case description="Should have a One Parameter" name="Demonstrate NUnit Problem.Should have a One Parameter" time="0.0165" asserts="0" success="True" result="Success" executed="True" />
</results>
</test-suite>
</results>
</test-suite>
</results>
</test-suite>
</test-results> This results in an error on line 18. (The last test-case)
|
I made a change to the output to fix the problem. I don't know if this is the way it should be fixed as I haven't looked in to NUnit much before this problem. <?xml version="1.0" encoding="utf-8" standalone="no"?>
<test-results xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="nunit_schema_2.5.xsd" name="Pester" total="4" errors="0" failures="0" not-run="0" inconclusive="0" ignored="0" skipped="0" invalid="0" date="2022-03-10" time="14:48:34">
<environment clr-version="4.0.30319.42000" user-domain="NA" cwd="C:\Temp\Test Pester NUnit Problem" platform="Microsoft Windows 10 Enterprise|C:\WINDOWS|\Device\Harddisk0\Partition3" machine-name="Computer" nunit-version="2.5.8.0" os-version="10.0.19043" user="User" />
<culture-info current-culture="en-GB" current-uiculture="en-GB" />
<test-suite type="TestFixture" name="Pester" executed="True" result="Success" success="True" time="0.1851" asserts="0" description="Pester">
<results>
<test-suite type="TestFixture" name="C:\Temp\Test Pester NUnit Problem\Test NUnit.ps1" executed="True" result="Success" success="True" time="0.1851" asserts="0" description="C:\Temp\Test Pester NUnit Problem\Test NUnit.ps1">
<results>
<test-suite type="TestFixture" name="Demonstrate NUnit Problem" executed="True" result="Success" success="True" time="0.0903" asserts="0" description="Demonstrate NUnit Problem">
<results>
<test-suite type="ParameterizedTest" name="Demonstrate NUnit Problem.Should return <Result> when type is <Type>" executed="True" result="Success" success="True" time="0.064" asserts="0" description="Should return <Result> when type is <Type>">
<results>
<test-case description="Should return True when type is String" name="Demonstrate NUnit Problem.Should return True when type is String" time="0.0362" asserts="0" success="True" result="Success" executed="True" />
<test-case description="Should return False when type is Int" name="Demonstrate NUnit Problem.Should return False when type is Int" time="0.0132" asserts="0" success="True" result="Success" executed="True" />
<test-case description="Should return False when type is Bool" name="Demonstrate NUnit Problem.Should return False when type is Bool" time="0.0146" asserts="0" success="True" result="Success" executed="True" />
</results>
</test-suite>
<test-suite type="TestFixture" name="Demonstrate NUnit Problem" executed="True" result="Success" success="True" time="0.0165" asserts="0" description="Demonstrate NUnit Problem">
<results>
<test-case description="Should have a One Parameter" name="Demonstrate NUnit Problem.Should have a One Parameter" time="0.0165" asserts="0" success="True" result="Success" executed="True" />
</results>
</test-suite>
</results>
</test-suite>
</results>
</test-suite>
</results>
</test-suite>
</test-results> |
I'm not seeing this in my tests, though I always have a Context-block that is doing the |
I see the same behaviour when mixing parameterised tests with standard tests inside a context block. Instead of a technical solution for this, it's probably simpler to advise to to keep these test types separate. <?xml version="1.0" encoding="utf-8" standalone="no"?>
<test-results xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="nunit_schema_2.5.xsd" name="Pester" total="4" errors="0" failures="0" not-run="0" inconclusive="0" ignored="0" skipped="0" invalid="0" date="2022-03-17" time="09:14:27">
<environment clr-version="4.0.30319.42000" user-domain="VERISK" cwd="C:\Temp\Test Pester NUnit Problem" platform="Microsoft Windows 10 Enterprise|C:\WINDOWS|\Device\Harddisk0\Partition3" machine-name="LV5KFXCB3" nunit-version="2.5.8.0" os-version="10.0.19043" user="I81052" />
<culture-info current-culture="en-GB" current-uiculture="en-GB" />
<test-suite type="TestFixture" name="Pester" executed="True" result="Success" success="True" time="5.7112" asserts="0" description="Pester">
<results>
<test-suite type="TestFixture" name="C:\Temp\Test Pester NUnit Problem\Test NUnit.ps1" executed="True" result="Success" success="True" time="5.7112" asserts="0" description="C:\Temp\Test Pester NUnit Problem\Test NUnit.ps1">
<results>
<test-suite type="TestFixture" name="Demonstrate NUnit Problem" executed="True" result="Success" success="True" time="2.612" asserts="0" description="Demonstrate NUnit Problem">
<results>
<test-suite type="TestFixture" name="Demonstrate NUnit Problem.Test NUnit" executed="True" result="Success" success="True" time="2.4491" asserts="0" description="Demonstrate NUnit Problem.Test NUnit">
<results>
<test-suite type="ParameterizedTest" name="Demonstrate NUnit Problem.Test NUnit.Should return <Result> when type is <Type>" executed="True" result="Success" success="True" time="1.6963" asserts="0" description="Should return <Result> when type is <Type>">
<results>
<test-case description="Should return True when type is String" name="Demonstrate NUnit Problem.Test NUnit.Should return True when type is String" time="1.6727" asserts="0" success="True" result="Success" executed="True" />
<test-case description="Should return False when type is Int" name="Demonstrate NUnit Problem.Test NUnit.Should return False when type is Int" time="0.0138" asserts="0" success="True" result="Success" executed="True" />
<test-case description="Should return False when type is Bool" name="Demonstrate NUnit Problem.Test NUnit.Should return False when type is Bool" time="0.0097" asserts="0" success="True" result="Success" executed="True" />
</results>
</test-suite>
<test-case description="Should have a One Parameter" name="Demonstrate NUnit Problem.Test NUnit.Should have a One Parameter" time="0.0345" asserts="0" success="True" result="Success" executed="True" />
</results>
</test-suite>
</results>
</test-suite>
</results>
</test-suite>
</results>
</test-suite>
</test-results> |
Good catch. Even the official NUnit packages like NUnit.Extension.NUnitV2ResultWriter for nunit3-console does it like Pester and doesn't comply with the resultsType-choice in the schema. Not sure if/when we will fix this. Working on NUnit3-support which allows combined @rcarpenter79 Is this blocking you in any way or just something you noticed? Every tool I've tested the reports with doesn't complain. They probably just use xpath to look for the |
Validating NUnit3 format is unaffected by pester#2143
General summary of the issue
When I mix tests in the same block where some contain data sets in ForEach with other tests that do not, the NUnit report is generated incorrectly and fails the NUnit2.5 schema validation.
Describe your environment
Steps to reproduce
I created a simple test that shows this problem.
I'm running this in VSCode.
I downloaded the NUnit2.5 xsd schema file and placed that in the same folder with the report and used Notepad++ with the XML Tools plugin to validate the report structure.
Possible Solution? (optional)
A work around is to separate tests in to different Context blocks
The text was updated successfully, but these errors were encountered: