Description
openedon Oct 7, 2021
Description
We are using the VSTest@2 task in azure DevOps for executing the test cases which having data rows in the test method. When we run the pipeline. It executes the only one data row that is executing. (Note: VSTest@2 task with testSelector: 'testPlan'). This issue has occurred on MSTest.TestAdapter & MSTest.TestFramework version 2.2.4 onwards. It is working on the 2.2.3 version.
- task: VSTest@2
displayName: 'Test using test plans'
inputs:
testSelector: 'testPlan'
testPlan: '23'
testSuite: '24'
testConfiguration: '3'
searchFolder: '$(System.DefaultWorkingDirectory)'
codeCoverageEnabled: true
platform: '$(BuildPlatform)'
configuration: '$(BuildConfiguration)'
enabled: true
continueOnError: true
Steps to reproduce
using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace TestProject1
{
[TestClass]
public class UnitTest1
{
[DataTestMethod]
[DataRow(new[] { "Connector_1", "Connector_2", "Connector_3", "Connector_4", "Connector_5", "Connector_6", "Connector_7", "Connector_8", "Connector_9", "Connector_10" })]
[DataRow(new[] { "Connector_5", "Connector_6", "Connector_7", "Connector_8", "Connector_9", "Connector_10" })]
[DataRow(new[] { "Connector_1", "Connector_2", "Connector_3", "Connector_4" })]
[DataRow(new[] { "Connector_2", "Connector_4", "Connector_6", "Connector_8", "Connector_10" })]
public void TestMethod3( string[] b)
{
Assert.AreEqual(b.Length, 4);
}
}
}
When we execute using MSTest.TestAdapter & MSTest.TestFramework version 2.2.6
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.11.0" />
<PackageReference Include="MSTest.TestAdapter" Version="2.6.0" />
<PackageReference Include="MSTest.TestFramework" Version="2.6.0" />
It was not reporting all data rows to test execution. See below the result which is executed into the pipeline
Actual behavior
**************** Starting test execution *********************
Using new test platform for test execution
Microsoft.TestPlatform.VsTestConsole.TranslationLayer.VsTestConsoleWrapper instance will be created
Failed TestProject1.UnitTest1.TestMethod3(System.String[]) TestMethod3 (System.String[])
Adding trx file D:\a\_temp\TestResults\VssAdministrator_WIN-O1PP9NTUSN3_2021-10-07_07_58_00.trx to run attachments
**************** Completed test execution *********************
Expected behavior
**************** Starting test execution *********************
Using new test platform for test execution
Microsoft.TestPlatform.VsTestConsole.TranslationLayer.VsTestConsoleWrapper instance will be created
Failed TestProject1.UnitTest1.TestMethod3(System.String[]) TestMethod3
Failed TestProject1.UnitTest1.TestMethod3(System.String[]) TestMethod3 (System.String[])
Failed TestProject1.UnitTest1.TestMethod3(System.String[]) TestMethod3 (System.String[])
Passed TestProject1.UnitTest1.TestMethod3(System.String[]) TestMethod3 (System.String[])
Failed TestProject1.UnitTest1.TestMethod3(System.String[]) TestMethod3 (System.String[])
Adding trx file D:\a\_temp\TestResults\VssAdministrator_WIN-O1PP9NTUSN3_2021-10-07_08_02_50.trx to run attachments
**************** Completed test execution *********************
If we use MSTest.TestAdapter & MSTest.TestFramework version 2.2.3 then it working as expected it reported four data rows. Please see the above result for the execution in Azure Pipeline.
Environment
Please share additional details about the test environment.
- Operating system: Agent (Pool:Azure Pipelines & Image: windows-latest)
- Build version of vstest.console: latest version
- Package version of MSTest framework and adapter: 2.2.6 for MSTest framework and adapter
- Other installed packages and their versions on the test project