The NUnit report is invalid when some tests contain ForEach/TestCase data

[rcarpenter79]

General summary of the issue

When I mix tests in the same block where some contain data sets in ForEach with other tests that do not, the NUnit report is generated incorrectly and fails the NUnit2.5 schema validation.

Describe your environment

Pester version     : 5.3.1 C:\Program Files\WindowsPowerShell\Modules\Pester\5.3.1\Pester.psm1
PowerShell version : 7.2.1
OS version         : Microsoft Windows NT 10.0.19043.0

Steps to reproduce

I created a simple test that shows this problem.
I'm running this in VSCode.

I downloaded the NUnit2.5 xsd schema file and placed that in the same folder with the report and used Notepad++ with the XML Tools plugin to validate the report structure.

function Test-IsString
{
    param
    (
        [PSObject]
        $Variable
    )

    return $Variable -is [String]
}

$PesterPreference = [PesterConfiguration]::Default
$PesterPreference.Output.Verbosity = 'Detailed'
$PesterPreference.TestResult.Enabled = $true

Push-Location -Path $PSScriptRoot

Describe -Name 'Demonstrate NUnit Problem' -Fixture {
    It 'Should return <Result> when type is <Type>' -ForEach @(
        @{ Type = 'String'; Variable = [String] 'Test'; Result = $true  }
        @{ Type = 'Int';    Variable = [Int] 0;         Result = $false }
        @{ Type = 'Bool';   Variable = [Bool] $true;    Result = $false }
    ) {
        Test-IsString -Variable $Variable | Should -Be $Result
    }

    It 'Should have a One Parameter' {
        (Get-Command Test-IsString).Parameters.Keys.Count | Should -Be 1
    }
}

Pop-Location

Possible Solution? (optional)

A work around is to separate tests in to different Context blocks

[johlju]

I have not seen this impact code coverage upload too tools like CodeCov.io. 🤔 Do you have a xml file that is generated erroneous and what the line should actually be? 🤔

[rcarpenter79]

Sure.

Here's the output from the code above.

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<test-results xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="nunit_schema_2.5.xsd" name="Pester" total="4" errors="0" failures="0" not-run="0" inconclusive="0" ignored="0" skipped="0" invalid="0" date="2022-03-10" time="14:48:34">
  <environment clr-version="4.0.30319.42000" user-domain="NA" cwd="C:\Temp\Test Pester NUnit Problem" platform="Microsoft Windows 10 Enterprise|C:\WINDOWS|\Device\Harddisk0\Partition3" machine-name="Computer" nunit-version="2.5.8.0" os-version="10.0.19043" user="User" />
  <culture-info current-culture="en-GB" current-uiculture="en-GB" />
  <test-suite type="TestFixture" name="Pester" executed="True" result="Success" success="True" time="0.1851" asserts="0" description="Pester">
    <results>
      <test-suite type="TestFixture" name="C:\Temp\Test Pester NUnit Problem\Test NUnit.ps1" executed="True" result="Success" success="True" time="0.1851" asserts="0" description="C:\Temp\Test Pester NUnit Problem\Test NUnit.ps1">
        <results>
          <test-suite type="TestFixture" name="Demonstrate NUnit Problem" executed="True" result="Success" success="True" time="0.0903" asserts="0" description="Demonstrate NUnit Problem">
            <results>
              <test-suite type="ParameterizedTest" name="Demonstrate NUnit Problem.Should return &lt;Result&gt; when type is &lt;Type&gt;" executed="True" result="Success" success="True" time="0.064" asserts="0" description="Should return &lt;Result&gt; when type is &lt;Type&gt;">
                <results>
                  <test-case description="Should return True when type is String" name="Demonstrate NUnit Problem.Should return True when type is String" time="0.0362" asserts="0" success="True" result="Success" executed="True" />
                  <test-case description="Should return False when type is Int" name="Demonstrate NUnit Problem.Should return False when type is Int" time="0.0132" asserts="0" success="True" result="Success" executed="True" />
                  <test-case description="Should return False when type is Bool" name="Demonstrate NUnit Problem.Should return False when type is Bool" time="0.0146" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
              <test-case description="Should have a One Parameter" name="Demonstrate NUnit Problem.Should have a One Parameter" time="0.0165" asserts="0" success="True" result="Success" executed="True" />
            </results>
          </test-suite>
        </results>
      </test-suite>
    </results>
  </test-suite>
</test-results>

This results in an error on line 18. (The last test-case)

Element 'test-case' is unexpected according to content model of parent element 'results'. Expecting: test-suite.

[rcarpenter79]

I made a change to the output to fix the problem.
The XML below validates as having the correct schema now.

I don't know if this is the way it should be fixed as I haven't looked in to NUnit much before this problem.
This might well cause problems somewhere else.

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<test-results xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="nunit_schema_2.5.xsd" name="Pester" total="4" errors="0" failures="0" not-run="0" inconclusive="0" ignored="0" skipped="0" invalid="0" date="2022-03-10" time="14:48:34">
  <environment clr-version="4.0.30319.42000" user-domain="NA" cwd="C:\Temp\Test Pester NUnit Problem" platform="Microsoft Windows 10 Enterprise|C:\WINDOWS|\Device\Harddisk0\Partition3" machine-name="Computer" nunit-version="2.5.8.0" os-version="10.0.19043" user="User" />
  <culture-info current-culture="en-GB" current-uiculture="en-GB" />
  <test-suite type="TestFixture" name="Pester" executed="True" result="Success" success="True" time="0.1851" asserts="0" description="Pester">
    <results>
      <test-suite type="TestFixture" name="C:\Temp\Test Pester NUnit Problem\Test NUnit.ps1" executed="True" result="Success" success="True" time="0.1851" asserts="0" description="C:\Temp\Test Pester NUnit Problem\Test NUnit.ps1">
        <results>
          <test-suite type="TestFixture" name="Demonstrate NUnit Problem" executed="True" result="Success" success="True" time="0.0903" asserts="0" description="Demonstrate NUnit Problem">
            <results>
              <test-suite type="ParameterizedTest" name="Demonstrate NUnit Problem.Should return &lt;Result&gt; when type is &lt;Type&gt;" executed="True" result="Success" success="True" time="0.064" asserts="0" description="Should return &lt;Result&gt; when type is &lt;Type&gt;">
                <results>
                  <test-case description="Should return True when type is String" name="Demonstrate NUnit Problem.Should return True when type is String" time="0.0362" asserts="0" success="True" result="Success" executed="True" />
                  <test-case description="Should return False when type is Int" name="Demonstrate NUnit Problem.Should return False when type is Int" time="0.0132" asserts="0" success="True" result="Success" executed="True" />
                  <test-case description="Should return False when type is Bool" name="Demonstrate NUnit Problem.Should return False when type is Bool" time="0.0146" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
              <test-suite type="TestFixture" name="Demonstrate NUnit Problem" executed="True" result="Success" success="True" time="0.0165" asserts="0" description="Demonstrate NUnit Problem">
                <results>
                  <test-case description="Should have a One Parameter" name="Demonstrate NUnit Problem.Should have a One Parameter" time="0.0165" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
            </results>
          </test-suite>
        </results>
      </test-suite>
    </results>
  </test-suite>
</test-results>

[johlju]

I'm not seeing this in my tests, though I always have a Context-block that is doing the ForEach and always have It-blocks inside Context-blocks. Can you confirm that it outputs correctly when wrapping inside a Context-block? If so then it just a problem when It-block is inside the Describe-block. 🤔

[rcarpenter79]

I see the same behaviour when mixing parameterised tests with standard tests inside a context block.
If you're doing foreach at the context block level this would naturally separate these test types.

Instead of a technical solution for this, it's probably simpler to advise to to keep these test types separate.
This is what I did to work around the problem I was seeing once I found out what was causing it.

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<test-results xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="nunit_schema_2.5.xsd" name="Pester" total="4" errors="0" failures="0" not-run="0" inconclusive="0" ignored="0" skipped="0" invalid="0" date="2022-03-17" time="09:14:27">
  <environment clr-version="4.0.30319.42000" user-domain="VERISK" cwd="C:\Temp\Test Pester NUnit Problem" platform="Microsoft Windows 10 Enterprise|C:\WINDOWS|\Device\Harddisk0\Partition3" machine-name="LV5KFXCB3" nunit-version="2.5.8.0" os-version="10.0.19043" user="I81052" />
  <culture-info current-culture="en-GB" current-uiculture="en-GB" />
  <test-suite type="TestFixture" name="Pester" executed="True" result="Success" success="True" time="5.7112" asserts="0" description="Pester">
    <results>
      <test-suite type="TestFixture" name="C:\Temp\Test Pester NUnit Problem\Test NUnit.ps1" executed="True" result="Success" success="True" time="5.7112" asserts="0" description="C:\Temp\Test Pester NUnit Problem\Test NUnit.ps1">
        <results>
          <test-suite type="TestFixture" name="Demonstrate NUnit Problem" executed="True" result="Success" success="True" time="2.612" asserts="0" description="Demonstrate NUnit Problem">
            <results>
              <test-suite type="TestFixture" name="Demonstrate NUnit Problem.Test NUnit" executed="True" result="Success" success="True" time="2.4491" asserts="0" description="Demonstrate NUnit Problem.Test NUnit">
                <results>
                  <test-suite type="ParameterizedTest" name="Demonstrate NUnit Problem.Test NUnit.Should return &lt;Result&gt; when type is &lt;Type&gt;" executed="True" result="Success" success="True" time="1.6963" asserts="0" description="Should return &lt;Result&gt; when type is &lt;Type&gt;">
                    <results>
                      <test-case description="Should return True when type is String" name="Demonstrate NUnit Problem.Test NUnit.Should return True when type is String" time="1.6727" asserts="0" success="True" result="Success" executed="True" />
                      <test-case description="Should return False when type is Int" name="Demonstrate NUnit Problem.Test NUnit.Should return False when type is Int" time="0.0138" asserts="0" success="True" result="Success" executed="True" />
                      <test-case description="Should return False when type is Bool" name="Demonstrate NUnit Problem.Test NUnit.Should return False when type is Bool" time="0.0097" asserts="0" success="True" result="Success" executed="True" />
                    </results>
                  </test-suite>
                  <test-case description="Should have a One Parameter" name="Demonstrate NUnit Problem.Test NUnit.Should have a One Parameter" time="0.0345" asserts="0" success="True" result="Success" executed="True" />
                </results>
              </test-suite>
            </results>
          </test-suite>
        </results>
      </test-suite>
    </results>
  </test-suite>
</test-results>

[fflaten]

Good catch. Even the official NUnit packages like NUnit.Extension.NUnitV2ResultWriter for nunit3-console does it like Pester and doesn't comply with the resultsType-choice in the schema. Not sure if/when we will fix this.

Working on NUnit3-support which allows combined test-suite and test-case under the same test-suite. Seeing similar issues there, were the official tools doesn't comply with their own schemas. 😕

@rcarpenter79 Is this blocking you in any way or just something you noticed? Every tool I've tested the reports with doesn't complain. They probably just use xpath to look for the test-case-elements.