Known Test Repository Issue Sometimes Misses Test Failures

[nagilson]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1196220

Build leg reported

Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewSearchTests.BasicTest

Pull Request

dotnet/sdk#51591

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The SSL connection",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

So, I believe there is an issue with the known issue flagger, where it is failing to mark certain failures as known, even if they are a known issue. Please let me know if I am mistaken.

Here's the issue:
Looking at this image, this test was not considered a known issue:
https://github.com/dotnet/sdk/pull/51591/checks?check_run_id=54740778719

I believe that, because it's showing as a test failure, and other similar test failures are counted in the above section:

Furthermore, it appeared to be blocking build-analysis from being green.

Here's such an example of 1 of several 'known issues' that flag this issue:
#5015
dotnet/sdk#51574
#6410

As you can see in, #51574 looks for the string "The SSL connection could not be established" as a ErrorMessage which is supposed to use String.Contains. The test, however, that I think was not flagged as a 'known issue', did indeed have that error message string. It looks like some were flagged, while others were not. Here's the output of that 'unknown' test failure:

Expected command to exit with 0 but it did not.\nFile Name: /Users/runner/work/1/s/artifacts/bin/redist/Release/dotnet/dotnet\nArguments: new console --search --debug:custom-hive /Users/runner/work/1/s/artifacts/tmp/Release/dotnet-new.IntegrationTests/SharedHomeDirectory/20251106211951401\nExit Code: 103\nStdOut:\nWarning: use of 'dotnet new --search' is deprecated. Use 'dotnet new search' instead.\nFor more information, run: \n   dotnet new search -h\n\nSearching for the templates...\nStdErr:\nMatches from template source: NuGet.org\nSearch failed: Failed to update search cache. (The SSL connection could not be established, see inner exception.) (The SSL connection could not be established, see inner exception.)\n\nFor details on the exit code, refer to https://aka.ms/templating-exit-codes#103\n [[Full log]](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1196220&view=ms.vss-test-web.build-test-results-tab&runId=33106418&resultId=100166&paneView=debug)

While there may be a confounding variable in that there are now several known issues which are tracking substrings of each other, or may overlap, the whole reason there are multiple in the first place is because the first existing known test repository issue did not flag all of the tests which failed with the known issue.

Could this alternatively be a limitation of:

1. A limit of how many tests can be flagged as a known issue for the same known issue
2. A different 'failing configuration' which causes issues to be missed?
3. A misunderstanding of which strings get searched? Note that I believe the error is also in the 'helix logs'.
4. A failure of my own accord to file the known issue? @dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1196220
Error message validated: [The SSL Connection C]
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 11/6/2025 10:39:11 PM UTC

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1196220
Error message validated: [The SSL connection]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 11/6/2025 10:42:09 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1196423	dotnet/machinelearning	Build	Log	dotnet/machinelearning#7543
1196398	dotnet/aspnetcore	Run build.sh	Log	dotnet/aspnetcore#64262
1196235	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121356
1196186	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#121417
1196168	dotnet/runtime	Send tests to Helix (Unix)	Log

Build	Definition	Test	Pull Request
1196358	dotnet/sdk	Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewSearchTests.CanShowDeprecationMessage_WhenLegacyCommandIsUsed	dotnet/sdk#51558
1196220	dotnet/sdk	Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewSearchTests.CanShowDeprecationMessage_WhenLegacyCommandIsUsed	dotnet/sdk#51591

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
7	7	7

Report

Build	Repository	Step Name	Console log	Pull Request
1537425	dotnet/runtime	Copy native test components to test output folder	Log	dotnet/runtime#131309
1535384	dotnet/machinelearning	Build	Log	dotnet/machinelearning#7669
3035268	dotnet-maui	Run Integration Tests - RunOniOS_MauiReleaseTrimFull_CoreCLR	Log
3034486	dotnet-aspnetcore	Publish Build Assets	Log
3032704	dotnet-maui	Run Integration Tests - Blazor	Log
1525487	dotnet/sdk	🟣 Build	Log	dotnet/sdk#55441
1523876	dotnet/runtime	Build Tests	Log	dotnet/runtime#131024
1523184	dotnet/maui	Controls Lifecycle,ManualReview,Maps	Log	dotnet/maui#36109
1517678	dotnet/aspnetcore	Run build.sh	Log	dotnet/aspnetcore#67900
3025210	dotnet-dotnet	Build	Log
1514833	dotnet/sdk	🟣 Build	Log	dotnet/sdk#55297
3024926	dotnet-dotnet	Build (azurelinuxX64CrossContainer)	Log
1512350	dotnet/sdk	Build	Log
1510295	dotnet/runtime	Build product	Log	dotnet/runtime#130775
1507550	dotnet/aspnetcore	Run build.sh	Log	dotnet/aspnetcore#67771
3016918	dotnet-maui	Run Integration Tests - RunOniOS_MauiRelease_CoreCLR	Log
1497250	dotnet/runtime	Build product	Log	dotnet/runtime#130300
1497101	dotnet/runtime	Build native test components	Log	dotnet/runtime#130299
3015617	dotnet-sdk	🟣 Build	Log
1495133	dotnet/maui	Install dotnet preview without workloads	Log	dotnet/maui#36334

Build	Repository	Test	Pull Request
1538838	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.AutomaticOrManual_DoesntFailRegardlessOfWhetherClientCertsAreAvailable
1538813	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1538638	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.AutomaticOrManual_DoesntFailRegardlessOfWhetherClientCertsAreAvailable
1537866	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1535640	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.AutomaticOrManual_DoesntFailRegardlessOfWhetherClientCertsAreAvailable
1534953	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1534774	dotnet/roslyn	Microsoft.CodeAnalysis.MSBuild.UnitTests.NewlyCreatedProjectsFromDotNetNew.ValidateCSharpTemplateProjects	dotnet/roslyn#84701
1534642	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1534639	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.AutomaticOrManual_DoesntFailRegardlessOfWhetherClientCertsAreAvailable
1534582	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1530310	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1527892	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1526361	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.AutomaticOrManual_DoesntFailRegardlessOfWhetherClientCertsAreAvailable
3030996	dotnet-sdk	Microsoft.NET.Pack.Tests.GivenThatThereAreImplicitPackageReferences.Microsoft.NET.Pack.Tests.GivenThatThereAreImplicitPackageReferences.Packing_a_netcoreapp_1_1_app_includes_the_implicit_dependency
1523325	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.AutomaticOrManual_DoesntFailRegardlessOfWhetherClientCertsAreAvailable
1521355	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1521470	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1521190	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1521102	dotnet/runtime	IntrinsicsInSystemPrivateCoreLib.Tests.WorkItemExecution
1520937	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.AutomaticOrManual_DoesntFailRegardlessOfWhetherClientCertsAreAvailable
1520451	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1519700	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1518979	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
3027306	dotnet-sdk	Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewInstallTests.ReinstallDoesntRemoveTemplates	#62654
1518559	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.AutomaticOrManual_DoesntFailRegardlessOfWhetherClientCertsAreAvailable
1517929	dotnet/sdk	CanShowMessageInCaseShortNameConflict
3025313	dotnet-sdk	Microsoft.NET.Build.Containers.UnitTests.dll.2.WorkItemExecution	#63003
1515623	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1515072	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1515081	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1514458	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1510254	dotnet/runtime	IntrinsicsInSystemPrivateCoreLib.Tests.WorkItemExecution
1508374	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1499665	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1502635	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1502303	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1501202	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1500024	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1499408	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1499115	dotnet/runtime	System.Net.Http.Functional.Tests.HttpClientHandler_RemoteServerTest.GetAsync_SetAutomaticDecompression_ContentDecompressed_Deflate
1499116	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_DangerousAcceptAllCertificatesValidator_Test.InvalidCertificateServers_CertificateValidationDisabled_Succeeds
1499111	dotnet/runtime	System.Net.Http.Functional.Tests.HttpClientHandler_RemoteServerTest.GetAsync_SetAutomaticDecompression_ContentDecompressed_Deflate
3016252	dotnet-sdk	Microsoft.NET.Build.Containers.UnitTests.dll.2.WorkItemExecution	#62654
1495952	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1495180	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1495192	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
4	15	66

[nagilson]

I'm still seeing this issue where the string contains match is identical, yet the known issue labeler doesn't think it's a known issue.