Test run is marked as failed even though all tests pass.

[rvarna]

Description

While running our tests in VSTS using the Test Task, we see that the test run is marked as failed although all tests pass. We also see "Error: An item with the same key has already been added" during test discovery, potentially causing the problem

Steps to reproduce

Reference MSTest and TestAdaper 1.2 Nuget packages. Run tests with test filter in VSTS using Test Task definition.

Expected behavior

Tests run is successful as all tests pass.

Actual behavior

Test run is marked as failed though all tests pass. Before the tests start executing we see a bunch of:

Warning: No test is available in Test0.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.

2017-10-25T06:00:23.9975712Z Warning: No test is available in Test1.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.

2017-10-25T06:00:24.0600773Z Warning: No test is available in Test2.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.

2017-10-25T06:00:24.3257236Z Error: An item with the same key has already been added.
...
...
2017-10-25T06:09:44.9663699Z Total tests: 39. Passed: 39. Failed: 0. Skipped: 0.
2017-10-25T06:09:44.9663699Z Test Run Failed.
2017-10-25T06:09:44.9663699Z Test execution time: 9.4706 Minutes
2017-10-25T06:09:45.0445030Z Results File: C:\BA\r1\a\TestResults\TestResults.trx
2017-10-25T06:09:45.3883132Z ##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
2017-10-25T06:09:45.6539600Z ##[error]Error: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe failed with return code: 1
2017-10-25T06:09:46.9822206Z ##[error]Vstest exited with return code: 2.

Environment

The tests are being in VSTS using the Test Task definition.
The test target .Net framework 4.6.2.
We run the tests based on test filters.

[AbhitejJohn]

@rvarna : Can you share diagnostic build logs with us please?

To get diag logs from the CI pipeline, you can set System.Debug in the Build definition to true.

If you could repro this locally using vstest.console, then it would also help to enable vstest.executionengine*.exe.config logging. Here is how you can enable that.

[rvarna]

@AbhitejJohn I forgot to mention that we are running these tests as part of the VSTS release definition. These are function tests but that shouldn't affect the behavior of vstest right? Would setting the variable System.Debug in build definition still help? Is the test adapter nuget required when running test using vstestconsole?

[rvarna]

Also, I played around with various settings for the task. Disabling "Run tests in parallel on multi-core machines" seems to do the trick and things work fine. Is there a non-thread safe data structure being used somewhere?

[AbhitejJohn]

That's odd, could you share the log file with parallel enabled with us please? Yes, System.Debug to true should still be applicable for Release Definitions.

[sudiptadmsft]

Unfortunately, right now we don't have enough information to investigate this issue further and find a solution. If this is still an issue for you, please update to our latest version. If you are still able to repro it, please provide us with more info!.