Skip Flakey GivenDotnetTest Tests

[nagilson]

In this PR a few days ago I disabled some similar tests: #46924

It turns out that even more of them are failing about 80%+ of the time after some recent changes to these tests.

See here: #47003 (review) here: #47004 (review) here: https://github.com/dotnet/sdk/pull/47004/checks?check_run_id=37586911841 here: #47014
etc.

I've already made an issue to track this: #46923 So I'll keep using that one. The maintainer has been notified.

[Copilot]

Copilot reviewed 1 out of 1 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (4)

test/dotnet-test.Tests/GivenDotnetTestBuildsAndRunsTestsForMultipleTFMs.cs:17

• [nitpick] Consider adding a brief description to the skip message for clarity, e.g., "Skipping due to flakiness: GivenDotnetTest Failures #46923".

[Theory(Skip= "https://github.com/dotnet/sdk/issues/46923")]

test/dotnet-test.Tests/GivenDotnetTestBuildsAndRunsTestsForMultipleTFMs.cs:56

• [nitpick] Consider adding a brief description to the skip message for clarity, e.g., "Skipping due to flakiness: GivenDotnetTest Failures #46923".

[Theory(Skip= "https://github.com/dotnet/sdk/issues/46923")]

test/dotnet-test.Tests/GivenDotnetTestBuildsAndRunsTestsForMultipleTFMs.cs:103

• [nitpick] Consider adding a brief description to the skip message for clarity, e.g., "Skipping due to flakiness: GivenDotnetTest Failures #46923".

[Theory(Skip= "https://github.com/dotnet/sdk/issues/46923")]

test/dotnet-test.Tests/GivenDotnetTestBuildsAndRunsTestsForMultipleTFMs.cs:131

• [nitpick] Consider adding a brief description to the skip message for clarity, e.g., "Skipping due to flakiness: GivenDotnetTest Failures #46923".

[Theory(Skip= "https://github.com/dotnet/sdk/issues/46923")]

[MiYanni]

Looks good. Always make sure to add the Test Debt label when making issues to skip tests. It allows us to keep track of these kinds of test-skipping issues.

[akoeplinger]

looks like we need to disable a couple more tests...

[mariam-abdulla]

I checked the tests, for the following tests we are already working on a fix:

DiscoverProjectWithMSTestMetaPackageAndMultipleTFMsWithTests_ShouldReturnExitCodeSuccess
RunProjectWithMSTestMetaPackageAndMultipleTFMs_ShouldReturnExitCodeGenericFailure

However, for the below tests I checked the old CI and they never actually failed:
RunMultipleProjectWithDifferentTFMs_ShouldReturnExitCodeGenericFailure
RunProjectWithMultipleTFMs_ShouldReturnExitCodeGenericFailure
RunProjectWithMultipleTFMsWithArchOption_ShouldReturnExitCodeGenericFailure

Please if you have the logs where they failed, let me know in which PRs to look.
Thank you!

[joeloff]

Tests like RunProjectWithMultipleTFMsWithArchOption_ShouldReturnExitCodeGenericFailure are failing because of the missing MSTest code coverage dependency. I just reran that locally and it passes once the missing feed is added. See #47095 and #47049

[mariam-abdulla]

Tests like RunProjectWithMultipleTFMsWithArchOption_ShouldReturnExitCodeGenericFailure are failing because of the missing MSTest code coverage dependency. I just reran that locally and it passes once the missing feed is added. See #47095 and #47049

Okay, what about #46924?
I fixed RunMultipleTestProjectsWithDifferentFailures_ShouldReturnExitCodeGenericFailure (still no PR yet) but for the rest, I couldn't see the old builds if they were failing.

[joeloff]

Some of these are related to the missing codecoverage package. I'm going to push changes to remove some of the skip attributes to verify that the missing dependency is resolved and then we should merge this I think

[joeloff]

Closing this in favor of #47095