Looking for help with upgrade to 3.9 - TestInitialize changed?

[davesmits]

Describe the bug

I am trying to upgrade to MSTest. This are the package changes I am now making

after upgrading some tests start to fail; it almost feels like objects created in the TestInitialize are not matched up with the right test (as if i get different variable values).

example

private Mock<MyObject> _mock;
[TestInitialize]
public async Task Initialize()
{
await Task.Delay(100); // just for demo
_mock = new Mock<MyObject>();
}

[TestMethod]
public async Taks ExecuteTestAsync()
{
_mock.Object.DoSomething();
}

When running my test the results are everything different with verifying the number of method invocations; Sometimes too much, sometimes not enough; but never consisted. When I add DoNotParallize it's already more stable but still 'random' results while these problems didnt exist before 3.9

Looking for some advice how to further diagnose this.

Steps To Reproduce

Looking for help to understand how to debug this?

Expected behavior

Test showing same before as version 3.8.3

Actual behavior

Test are flaky and unpredictable

[nohwnd]

Might be related to what we are fixing here: #5636

We're looking into it.

[davesmits]

@nohwnd is there already a pre release package of that, that i can give a try?

[Youssef1313]

So far I'm not able to reproduce the issue. If the codebase you're working with isn't open source and you cannot share it, are you able to create a new MSTest project, try to repro there, and upload that project?

[nohwnd]

Not yet, it just started building, should be on https://aka.ms/mstest/preview in ~2 hours. Will let you know.

[Evangelink]

@davesmits we are trying to to remove more burdden from our users (keeping packages in sync, updates deps...), by improving MSTest metapackage so that it offers more of the needed dependencies by default.

Whether you are using VSTest or MTP (MSTest runner), you should be able to drop MSTest.TestFramework and Microsoft.NET.Test.Sdk that are already dependencies of MSTest. If you are using MTP/MSTest runner, you should also be able to drop the MS CC dependency.

[davesmits]

@Youssef1313 yea it was more a conceptuel example; I am not sure how to exactly reproduce it. the project isn't open source or something that can share; also i think it brings way too much complicated stuff in. it's why asked for more guidence in to debug so could work towards a minimal reproduction

@nohwnd let me know and will give it a test!

@Evangelink thanks. yea we need to clean it up. that code already exists since .net core 2.1 so lot of things been touched

[nohwnd]

https://dev.azure.com/dnceng/public/_artifacts/feed/test-tools/NuGet/MSTest/overview/3.10.0-preview.25276.7

Yes, should be this one. See the aka link above for full guide how to use matest previews.

[davesmits]

@nohwnd seems to fix it! shall I clsoe the issue since there is already another

[Youssef1313]

Happy to know it's fixing it. I think this can be closed yes.