Skip to content

Test | Add flaky test quarantine zone #3856

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mdaigle wants to merge 9 commits into
base: main
Choose a base branch
from
+142 −23
Open

Test | Add flaky test quarantine zone #3856

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
fa22414
Skip flaky tests. Try removing platform filters which should already …
mdaigle Dec 19, 2025
dba39e7
Make filter configurable in build.proj
mdaigle Dec 19, 2025
13b684c
Add quarantined testing steps.
mdaigle Dec 19, 2025
8a4f8e0
Fix closing quote.
mdaigle Dec 19, 2025
4d14a1e
Fix closing quote.
mdaigle Dec 19, 2025
ae19c47
Fix quote wrapping for filter argument.
mdaigle Dec 19, 2025
29ecebc
Fix continueOnError indent.
mdaigle Dec 19, 2025
3413a96
Tag flaky tests.
mdaigle Dec 20, 2025
4c4cf1e
Tag xevent test as flaky.
mdaigle Dec 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 19 additions & 7 deletions build.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,8 @@
<!-- Run all unit tests applicable to Windows. -->
<Target Name="RunUnitTestsWindows" Condition="'$(IsEnabledWindows)' == 'true' AND $(ReferenceType.Contains('Project'))">
<PropertyGroup>
<FilterStatement>$(Filter)</FilterStatement>
<FilterStatement Condition="'$(FilterStatement)' == ''">category!=failing&amp;category!=flaky</FilterStatement>
<TestCommand>
$(DotnetPath)dotnet test "@(UnitTestsProj)"
--no-build
Expand All @@ -232,18 +234,20 @@
-p:TestTargetOS=Windows$(TargetGroup)
--collect "Code coverage"
--results-directory $(ResultsDirectory)
--filter "category!=failing%26category!=flaky"
--filter "$(FilterStatement)"
--logger:"trx;LogFilePrefix=Unit-Windows$(TargetGroup)-$(TestSet)"
</TestCommand>
<TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand>
</PropertyGroup>
<Message Text=">>> Running unit tests for Windows via command: $(TestCommand)"/>
<Exec ConsoleToMsBuild="true" Command="$(TestCommand)"/>

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net8_0_AnyCPU_NativeSNI_2)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category!=failing&category!=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Azure_Sql net462_AnyCPU_1)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetfxVersion=net462         -p:TestTargetOS=Windowsnetfx         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetfx-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Sql22 net9_0_AnyCPU_ManagedSNI_3)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Sql22 net8_0_AnyCPU_ManagedSNI_3)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Sql22_Named_Instance net9_0_AnyCPU_ManagedSNI_1)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Sql22_Named_Instance net9_0_AnyCPU_ManagedSNI_2)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net9_0_AnyCPU_NativeSNI_1)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net462_AnyCPU_1)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetfxVersion=net462         -p:TestTargetOS=Windowsnetfx         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetfx-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Sql19_x86 net8_0_AnyCPU_NativeSNI_3)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         C:\x86\dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net9_0_AnyCPU_NativeSNI_2)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Sql22_x86 net462_AnyCPU_3)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         C:\x86\dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetfxVersion=net462         -p:TestTargetOS=Windowsnetfx         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetfx-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net8_0_AnyCPU_NativeSNI_1)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net9_0_AnyCPU_ManagedSNI_2)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net8_0_AnyCPU_ManagedSNI_1)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net9_0_AnyCPU_ManagedSNI_1)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net8_0_AnyCPU_ManagedSNI_2)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net462_AnyCPU_2)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetfxVersion=net462         -p:TestTargetOS=Windowsnetfx         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetfx-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Sql19 net8_0_AnyCPU_ManagedSNI_2)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net8_0_AnyCPU_ManagedSNI_3)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net9_0_AnyCPU_ManagedSNI_3)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 243 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net8_0_AnyCPU_ManagedSNI_3)

build.proj#L243 

build.proj(243,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.
</Target>

<!-- Run all unit tests applicable to Unix. -->
<Target Name="RunUnitTestsUnix" Condition="'$(IsEnabledWindows)' != 'true' AND $(ReferenceType.Contains('Project'))">
<PropertyGroup>
<FilterStatement>$(Filter)</FilterStatement>
<FilterStatement Condition="'$(FilterStatement)' == ''">category!=failing&amp;category!=flaky</FilterStatement>
<TestCommand>
$(DotnetPath)dotnet test "@(UnitTestsProj)"
--no-build
Expand All @@ -253,7 +257,7 @@
-p:TestTargetOS=Unixnetcoreapp
--collect "Code coverage"
--results-directory $(ResultsDirectory)
--filter "category!=failing%26category!=flaky"
--filter "$(FilterStatement)"
--logger:"trx;LogFilePrefix=Unit-Unixnetcoreapp-$(TestSet)"
</TestCommand>
<TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand>
Expand All @@ -268,6 +272,8 @@
<!-- Run all Functional tests applicable to Windows. -->
<Target Name="RunFunctionalTestsWindows" Condition="'$(IsEnabledWindows)' == 'true'">
<PropertyGroup>
<FilterStatement>$(Filter)</FilterStatement>
<FilterStatement Condition="'$(FilterStatement)' == ''">category!=failing&amp;category!=flaky</FilterStatement>
<TestCommand>
$(DotnetPath)dotnet test "@(FunctionalTestsProj)"
--no-build
Expand All @@ -279,7 +285,7 @@
-p:TestTargetOS=Windows$(TargetGroup)
--collect "Code coverage"
--results-directory $(ResultsDirectory)
--filter "category!=non$(TargetGroup)tests&amp;category!=failing&amp;category!=nonwindowstests"
--filter "$(FilterStatement)"
Copy link

Copilot AI Dec 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The filter has been changed from "category!=non$(TargetGroup)tests&category!=failing&category!=nonwindowstests" to "category!=failing&category!=flaky". This removes important platform and framework-specific test exclusions. Tests tagged with 'non$(TargetGroup)tests' or 'nonwindowstests' will now run even though they aren't compatible with this configuration. The filter should be: "category!=non$(TargetGroup)tests&category!=failing&category!=nonwindowstests&category!=flaky"

Copilot uses AI. Check for mistakes.
--logger:"trx;LogFilePrefix=Functional-Windows$(TargetGroup)-$(TestSet)"
</TestCommand>
<TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand>
Expand All @@ -290,7 +296,9 @@

<!-- Run all Functional tests applicable to Unix. -->
<Target Name="RunFunctionalTestsUnix" Condition="'$(IsEnabledWindows)' != 'true'">
<PropertyGroup>
<PropertyGroup>
Copy link

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inconsistent indentation: The opening <PropertyGroup> tag has an extra space of indentation compared to other similar targets. It should be indented with 4 spaces to align with the same element in other targets like RunFunctionalTestsWindows at line 274.

Suggested change
<PropertyGroup>
<PropertyGroup>

Copilot uses AI. Check for mistakes.
<FilterStatement>$(Filter)</FilterStatement>
<FilterStatement Condition="'$(FilterStatement)' == ''">category!=failing&amp;category!=flaky</FilterStatement>
<TestCommand>
$(DotnetPath)dotnet test "@(FunctionalTestsProj)"
--no-build
Expand All @@ -302,7 +310,7 @@
-p:TestTargetOS=Unixnetcoreapp
--collect "Code coverage"
--results-directory $(ResultsDirectory)
--filter "category!=nonnetcoreapptests&amp;category!=failing&amp;category!=nonlinuxtests&amp;category!=nonuaptests"
--filter "$(FilterStatement)"
Copy link

Copilot AI Dec 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The filter has been changed from "category!=nonnetcoreapptests&category!=failing&category!=nonlinuxtests&category!=nonuaptests" to "category!=failing&category!=flaky". This removes important platform and framework-specific test exclusions. Tests tagged with 'nonnetcoreapptests', 'nonlinuxtests', or 'nonuaptests' will now run even though they aren't compatible with this configuration. The filter should be: "category!=nonnetcoreapptests&category!=failing&category!=nonlinuxtests&category!=nonuaptests&category!=flaky"

Copilot uses AI. Check for mistakes.
--logger:"trx;LogFilePrefix=Functional-Unixnetcoreapp-$(TestSet)"
</TestCommand>
<TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand>
Expand All @@ -317,6 +325,8 @@
<!-- Run all Manual tests applicable to Windows. -->
<Target Name="RunManualTestsWindows" Condition="'$(IsEnabledWindows)' == 'true'">
<PropertyGroup>
<FilterStatement>$(Filter)</FilterStatement>
<FilterStatement Condition="'$(FilterStatement)' == ''">category!=failing&amp;category!=flaky</FilterStatement>
<TestCommand>
$(DotnetPath)dotnet test "@(ManualTestsProj)"
--no-build
Expand All @@ -328,18 +338,20 @@
-p:TestTargetOS=Windows$(TargetGroup)
--collect "Code coverage"
--results-directory $(ResultsDirectory)
--filter "category!=non$(TargetGroup)tests&amp;category!=failing&amp;category!=nonwindowstests"
--filter "$(FilterStatement)"
Copy link

Copilot AI Dec 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The filter has been changed from "category!=non$(TargetGroup)tests&category!=failing&category!=nonwindowstests" to "category!=failing&category!=flaky". This removes important platform and framework-specific test exclusions. Tests tagged with 'non$(TargetGroup)tests' or 'nonwindowstests' will now run even though they aren't compatible with this configuration. The filter should be: "category!=non$(TargetGroup)tests&category!=failing&category!=nonwindowstests&category!=flaky"

Copilot uses AI. Check for mistakes.
--logger:"trx;LogFilePrefix=Manual-Windows$(TargetGroup)-$(TestSet)"
</TestCommand>
<TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand>
</PropertyGroup>
<Message Text=">>> Running Manual test for Windows via command: $(TestCommand)" />
<Exec ConsoleToMsBuild="true" Command="$(TestCommand)" />

Check failure on line 347 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Azure_Sql net8_0_AnyCPU_NativeSNI_3)

build.proj#L347 

build.proj(347,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:ReferenceType=Project         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 347 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Azure_Sql net8_0_AnyCPU_NativeSNI_3)

build.proj#L347 

build.proj(347,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:ReferenceType=Project         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 347 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Package (Win22_Azure_Sql net9_0_AnyCPU_ManagedSNI_3)

build.proj#L347 

build.proj(347,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:ReferenceType=Package         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 347 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net9_0_AnyCPU_NativeSNI_3)

build.proj#L347 

build.proj(347,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:ReferenceType=Project         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 347 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net9_0_AnyCPU_ManagedSNI_3)

build.proj#L347 

build.proj(347,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:ReferenceType=Project         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 347 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net462_AnyCPU_3)

build.proj#L347 

build.proj(347,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetfxVersion=net462         -p:ReferenceType=Project         -p:TestSet=3         -p:TestTargetOS=Windowsnetfx         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetfx-3"       " exited with code 1.

Check failure on line 347 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Azure_Sql net9_0_AnyCPU_ManagedSNI_3)

build.proj#L347 

build.proj(347,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:ReferenceType=Project         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category!=failing&category!=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 347 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net8_0_AnyCPU_NativeSNI_3)

build.proj#L347 

build.proj(347,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:ReferenceType=Project         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category!=failing&category!=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 347 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project

build.proj#L347 

build.proj(347,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:ReferenceType=Project         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.
</Target>

<!-- Run all Manual tests applicable to Unix. -->
<Target Name="RunManualTestsUnix" Condition="'$(IsEnabledWindows)' != 'true'">
<PropertyGroup>
<FilterStatement>$(Filter)</FilterStatement>
<FilterStatement Condition="'$(FilterStatement)' == ''">category!=failing&amp;category!=flaky</FilterStatement>
<TestCommand>
$(DotnetPath)dotnet test "@(ManualTestsProj)"
--no-build
Expand All @@ -351,7 +363,7 @@
-p:TestTargetOS=Unixnetcoreapp
--collect "Code coverage"
--results-directory $(ResultsDirectory)
--filter "category!=nonnetcoreapptests&amp;category!=failing&amp;category!=nonlinuxtests&amp;category!=nonuaptests"
--filter "$(FilterStatement)"
Copy link

Copilot AI Dec 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The filter has been changed from "category!=nonnetcoreapptests&category!=failing&category!=nonlinuxtests&category!=nonuaptests" to "category!=failing&category!=flaky". This removes important platform and framework-specific test exclusions. Tests tagged with 'nonnetcoreapptests', 'nonlinuxtests', or 'nonuaptests' will now run even though they aren't compatible with this configuration. The filter should be: "category!=nonnetcoreapptests&category!=failing&category!=nonlinuxtests&category!=nonuaptests&category!=flaky"

Copilot uses AI. Check for mistakes.
--logger:"trx;LogFilePrefix=Manual-Unixnetcoreapp-$(TestSet)"
</TestCommand>
<TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand>
Expand Down
21 changes: 19 additions & 2 deletions eng/pipelines/common/templates/steps/build-and-run-tests-netcore-step.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,12 +73,29 @@ steps:
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.SqlClient.FunctionalTests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=nonnetcoreapptests&category!=failing&category!=nonwindowstests"'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=failing&category!=flaky"'
Copy link

Copilot AI Dec 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The filter has been changed from excluding nonnetcoreapptests, failing, and nonwindowstests to only excluding failing and flaky tests. This means tests tagged with 'nonnetcoreapptests' or 'nonwindowstests' will now run in this pipeline, which could cause failures if those tests aren't compatible with .NET Core or Windows. The filter should maintain the original exclusions: "category!=nonnetcoreapptests&category!=failing&category!=nonwindowstests&category!=flaky"

Copilot uses AI. Check for mistakes.

- task: DotNetCoreCLI@2
displayName: 'Run Flaky Functional Tests for ${{parameters.TargetNetCoreVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.SqlClient.FunctionalTests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category=flaky"'
continueOnError: true

- task: DotNetCoreCLI@2
displayName: 'Run Manual Tests for ${{parameters.TargetNetCoreVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter category!=nonnetcoreapptests&category!=failing&category!=nonwindowstests --collect "Code Coverage"'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=failing&category!=flaky" --collect "Code Coverage"'
Copy link

Copilot AI Dec 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The filter has been changed from excluding nonnetcoreapptests, failing, and nonwindowstests to only excluding failing and flaky tests. This means tests tagged with 'nonnetcoreapptests' or 'nonwindowstests' will now run in this pipeline, which could cause failures if those tests aren't compatible with .NET Core or Windows. The filter should maintain the original exclusions: "category!=nonnetcoreapptests&category!=failing&category!=nonwindowstests&category!=flaky"

Copilot uses AI. Check for mistakes.
retryCountOnTaskFailure: ${{parameters.retryCountOnManualTests }}

- task: DotNetCoreCLI@2
displayName: 'Run Flaky Manual Tests for ${{parameters.TargetNetCoreVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category=flaky" --collect "Code Coverage"'
retryCountOnTaskFailure: ${{parameters.retryCountOnManualTests }}
continueOnError: true
21 changes: 19 additions & 2 deletions eng/pipelines/common/templates/steps/build-and-run-tests-netfx-step.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,12 +72,29 @@ steps:
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.SqlClient.FunctionalTests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=nonnetfxtests&category!=failing&category!=nonwindowstests" --collect "Code Coverage"'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=failing&category!=flaky" --collect "Code Coverage"'
Copy link

Copilot AI Dec 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The filter has been changed from excluding nonnetfxtests, failing, and nonwindowstests to only excluding failing and flaky tests. This means tests tagged with 'nonnetfxtests' or 'nonwindowstests' will now run in this pipeline, which could cause failures if those tests aren't compatible with .NET Framework or Windows. The filter should maintain the original exclusions: "category!=nonnetfxtests&category!=failing&category!=nonwindowstests&category!=flaky"

Copilot uses AI. Check for mistakes.

- task: DotNetCoreCLI@2
displayName: 'Run Flaky Functional Tests for ${{parameters.TargetNetFxVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.SqlClient.FunctionalTests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category=flaky" --collect "Code Coverage"'
continueOnError: true

- task: DotNetCoreCLI@2
displayName: 'Run Manual Tests for ${{parameters.TargetNetFxVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=nonnetfxtests&category!=failing&category!=nonwindowstests" --collect "Code Coverage"'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=failing&category!=flaky" --collect "Code Coverage"'
Copy link

Copilot AI Dec 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The filter has been changed from excluding nonnetfxtests, failing, and nonwindowstests to only excluding failing and flaky tests. This means tests tagged with 'nonnetfxtests' or 'nonwindowstests' will now run in this pipeline, which could cause failures if those tests aren't compatible with .NET Framework or Windows. The filter should maintain the original exclusions: "category!=nonnetfxtests&category!=failing&category!=nonwindowstests&category!=flaky"

Copilot uses AI. Check for mistakes.
retryCountOnTaskFailure: ${{parameters.retryCountOnManualTests }}

- task: DotNetCoreCLI@2
displayName: 'Run Flaky Manual Tests for ${{parameters.TargetNetFxVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category=flaky" --collect "Code Coverage"'
retryCountOnTaskFailure: ${{parameters.retryCountOnManualTests }}
continueOnError: true
Loading
Loading