Skip to content

Failures in SSL Stream tests: There are no more endpoints available in the endpoint mapper. #74838

Closed
@carlossanlop

Description

@carlossanlop

Occurrences from Runfo based on last 30 days and on Test Monitor history:

Day Run Failures Details
9/8 Rolling run 9900 (7.0) 2x entries net7.0-windows-Release-x86-CoreCLR_release-Windows.10.Amd64.Server2022.ES.Open - 2x System.Reflection.PortableExecutable.Tests.PEBuilderTests (BasicValidationSigned & Checksum)
9/7 Rolling run 6928 360x entries net7.0-windows-Release-x86-CoreCLR_release-Windows.10.Amd64.Server2022.ES.Open
8/24 PR 1964645 - PR #62863 1,016x failures in Kusto net7.0-windows-Debug-x64-CoreCLR_release-Windows.10.Amd64.Server2022.ES.Open
~8/11 PR 1936509 ? (data not available in Runfo) / PR not listed in Kusto net7.0-windows-Debug-x64-CoreCLR_release-Windows.10.Amd64.Server2022.ES.Open
8/6 Rolling run 1927354 828x failures / 1,269x failures in Kusto net7.0-windows-Release-x86-CoreCLR_release-Windows.10.Amd64.Server2022.ES.Open & net7.0-windows-Release-x64-CoreCLR_release-Windows.10.Amd64.Server2022.ES.Open
8/4 PR #73200 2,566x failures in Kusto Windows.10.Amd64.Server2022.ES.Open - x86 & x64
7/29 Rolling run 112x failures in Kusto Windows.10.Amd64.Server2022.ES.Open - x64
7/29 PR #73061 1,619x failures in Kusto Windows.10.Amd64.Server2022.ES.Open - x64 & x86
7/29 PR #73057 293x failures in Kusto Windows.10.Amd64.Server2022.ES.Open - x64 & x86
7/29 PR #73055 789x failures in Kusto Windows.10.Amd64.Server2022.ES.Open - x64
7/27 PR #72882 789x failures in Kusto Windows.10.Amd64.Server2022.ES.Open - x64
7/26 PR #72886 1,239x failures in Kusto Windows.10.Amd64.Server2022.ES.Open - x86 & x64
7/26 PR #72869 882x failures in Kusto Windows.10.Amd64.Server2022.ES.Open - x64
7/25 PR #72814 1,082x failures in Kusto Windows.10.Amd64.Server2022.ES.Open - x86 & x64
6/30 PR #71473 787x failures in Kusto Windows.10.Amd64.Server2022.ES.Open - x64
6/30 PR #71405 329x failures in Kusto Windows.10.Amd64.Server2022.ES.Open - x86 & x64

This happened in the release/7.0 branch. Can you please confirm if this will require a backported fix?

PR: #74808
Queue: Libraries Test Run release coreclr windows x86 Release
Job: https://dev.azure.com/dnceng/public/_build/results?buildId=1976097&view=logs&j=457f7e88-dfa2-5bd9-f871-fdf124c2477d&t=bfe52dfb-2099-5c7f-ee52-70a1d81c544e
Log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-74808-merge-0a29e7160d114b13be/System.Net.Security.Tests/3/console.38a99609.log?helixlogtype=result

Failure message:

CryptographicException : No hay más extremos disponibles desde el asignador de extremos.

Which translates to:

CryptographicException : There are no more endpoints available in the endpoint mapper.

System.Net.Security tests with this failure:

System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_SecondNegotiateClientCertificateAsync_Throws
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_UntrustedCaWithCustomTrust_OK
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsync_IncompleteIncomingTlsFrame_Throws
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsync_ClientWriteData
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsyncNoRenego_Succeeds
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NestedAuth_Throws
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsyncConcurrentIO_Throws
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsync_PendingDecryptedData_Throws
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsyncTls13_Succeeds
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NegotiateClientCertificateAsync_Succeeds
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_TargetHostName_Succeeds
System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_RandomSizeWrites_OK
System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback
System.Net.Security.Tests.SslStreamMutualAuthenticationTest.SslStream_RequireClientCert_IsMutuallyAuthenticated_ReturnsTrue
System.Net.Security.Tests.SslStreamCredentialCacheTest.SslStream_SameCertUsedForClientAndServer_Ok

System.Net.Http tests with this failure:

System.Net.Http.Functional.Tests.PlatformHandler_HttpClientHandler_ServerCertificates_Http2_Test.UseCallback_SelfSignedCertificate_ExpectedPolicyErrors
System.Net.Http.WinHttpHandlerFunctional.Tests.ClientCertificateTest.UseClientCertOnHttp2_DowngradedToHttp1MutualAuth_Success

Callstack example:

    System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_SecondNegotiateClientCertificateAsync_Throws(sendClientCertificate: True) [FAIL]
      System.AggregateException : One or more errors occurred. (No hay m�s extremos disponibles desde el asignador de extremos.) (The following constructor parameters did not have matching fixture data: CertificateSetup setup)
      ---- System.Security.Cryptography.CryptographicException : No hay m�s extremos disponibles desde el asignador de extremos.
      ---- The following constructor parameters did not have matching fixture data: CertificateSetup setup
      Stack Trace:
        
        ----- Inner Stack Trace #1 (System.Security.Cryptography.CryptographicException) -----
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CngHelpers.cs(33,0): at System.Security.Cryptography.CngHelpers.OpenStorageProvider(CngProvider provider)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CngKey.Create.cs(41,0): at System.Security.Cryptography.CngKey.Create(CngAlgorithm algorithm, String keyName, CngKeyCreationParameters creationParameters)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CngAlgorithmCore.cs(73,0): at System.Security.Cryptography.CngAlgorithmCore.GetOrGenerateKey(Int32 keySize, CngAlgorithm algorithm)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSACng.Key.cs(24,0): at System.Security.Cryptography.RSACng.get_Key()
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSACng.ImportExport.cs(50,0): at System.Security.Cryptography.RSACng.ExportKeyBlob(Boolean includePrivateParameters)
        /_/src/libraries/Common/src/System/Security/Cryptography/RSACng.ImportExport.cs(282,0): at System.Security.Cryptography.RSACng.ExportParameters(Boolean includePrivateParameters)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSA.cs(811,0): at System.Security.Cryptography.RSA.WritePkcs1PublicKey()
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSA.cs(675,0): at System.Security.Cryptography.RSA.ExportRSAPublicKey()
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSAWrapper.cs(141,0): at System.Security.Cryptography.RSAWrapper.ExportRSAPublicKey()
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/RSAPkcs1X509SignatureGenerator.cs(37,0): at System.Security.Cryptography.X509Certificates.RSAPkcs1X509SignatureGenerator.BuildPublicKey(RSA rsa)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/RSAPkcs1X509SignatureGenerator.cs(27,0): at System.Security.Cryptography.X509Certificates.RSAPkcs1X509SignatureGenerator.BuildPublicKey()
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/X509SignatureGenerator.cs(10,0): at System.Security.Cryptography.X509Certificates.X509SignatureGenerator.get_PublicKey()
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/CertificateRequest.cs(141,0): at System.Security.Cryptography.X509Certificates.CertificateRequest..ctor(String subjectName, RSA key, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding)
        /_/src/libraries/Common/tests/System/Security/Cryptography/X509Certificates/CertificateAuthority.cs(820,0): at System.Security.Cryptography.X509Certificates.Tests.Common.CertificateAuthority.BuildPrivatePki(PkiOptions pkiOptions, RevocationResponder& responder, CertificateAuthority& rootAuthority, CertificateAuthority[]& intermediateAuthorities, X509Certificate2& endEntityCert, Int32 intermediateAuthorityCount, String testName, Boolean registerAuthorities, Boolean pkiOptionsInSubject, String subjectName, Int32 keySize, X509ExtensionCollection extensions)
        /_/src/libraries/System.Net.Security/tests/FunctionalTests/TestHelper.cs(168,0): at System.Net.Security.Tests.TestHelper.GenerateCertificates(String targetName, String testName, Boolean longChain, Boolean serverCertificate)
        /_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamNetworkStreamTest.cs(30,0): at System.Net.Security.Tests.CertificateSetup..ctor()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/ConstructorInvoker.cs(86,0): at System.Reflection.ConstructorInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

Other exceptions:

System.Security.Cryptography.CryptographicException : Datos incorrectos.

   at System.Security.Cryptography.X509Certificates.CertificatePal.FilterPFXStore(ReadOnlySpan`1 rawData, SafePasswordHandle password, PfxCertStoreFlags pfxCertStoreFlags) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/CertificatePal.Windows.Import.cs:line 167
   at System.Security.Cryptography.X509Certificates.CertificatePal.FromBlobOrFile(ReadOnlySpan`1 rawData, String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/CertificatePal.Windows.Import.cs:line 88
   at System.Security.Cryptography.X509Certificates.CertificatePal.FromBlob(ReadOnlySpan`1 rawData, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/CertificatePal.Windows.Import.cs:line 16
   at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(Byte[] rawData, String password, X509KeyStorageFlags keyStorageFlags) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/X509Certificate.cs:line 95
   at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(Byte[] rawData, String password, X509KeyStorageFlags keyStorageFlags) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/X509Certificate2.cs:line 72
   at Test.Cryptography.CertLoader.TryGetCertificateWithPrivateKey(Boolean exportable) in /_/src/libraries/System.Security.Cryptography.Pkcs/tests/CertLoader.cs:line 84
   at System.Security.Cryptography.Pkcs.Tests.SignedCmsTests.SignWithImplicitSubjectKeyIdentifier() in /_/src/libraries/System.Security.Cryptography.Pkcs/tests/SignedCms/SignedCmsTests.cs:line 1225
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) in /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodInvoker.cs:line 64
System.Security.Cryptography.CryptographicException : Algoritmo especificado no es válido.

   at System.Security.Cryptography.CapiHelper.CreateHashHandle(SafeProvHandle hProv, Byte[] hash, Int32 calgHash, Boolean throwOnSizeError) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CapiHelper.Windows.cs:line 1385
   at System.Security.Cryptography.CapiHelper.CreateHashHandle(SafeProvHandle hProv, Byte[] hash, Int32 calgHash) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CapiHelper.Windows.cs:line 1373
   at System.Security.Cryptography.CapiHelper.SignValue(SafeProvHandle hProv, SafeCapiKeyHandle hKey, Int32 keyNumber, Int32 calgKey, Int32 calgHash, Byte[] hash) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CapiHelper.Windows.cs:line 1161
   at System.Security.Cryptography.RSACryptoServiceProvider.SignHash(Byte[] rgbHash, Int32 calgHash) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSACryptoServiceProvider.Windows.cs:line 504
   at System.Security.Cryptography.RSACryptoServiceProvider.SignHash(Byte[] hash, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSACryptoServiceProvider.Windows.cs:line 633
   at System.Security.Cryptography.RSA.SignData(Byte[] data, Int32 offset, Int32 count, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSA.cs:line 189
   at System.Security.Cryptography.RSA.SignData(Byte[] data, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSA.cs:line 170
   at System.Security.Cryptography.X509Certificates.Tests.CollectionTests.SerializedCertDisposeDoesNotRemoveKeyFile() in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CollectionTests.cs:line 1437
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) in /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodInvoker.cs:line 64
System.TypeInitializationException : The type initializer for 'System.Security.Cryptography.CngKeyLite' threw an exception.
---- Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException : No hay más extremos disponibles desde el asignador de extremos.
let failedTests = (testNameSubstring : string, methodName : string, messageSubstr: string, includePR : bool, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    //| where Message contains messageSubstr
    | where (Message contains 'System.Security.Cryptography.CryptographicException : No hay más extremos disponibles desde el asignador de extremos.' or Message contains 'System.Security.Cryptography.CryptographicException : Datos incorrectos.' or Message contains 'System.Security.Cryptography.CryptographicException : Algoritmo especificado no es válido.')
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        //| where Branch == 'refs/pull//merge'
        //| where Branch == 'refs/pull//merge'
        | where Branch != 'refs/pull/71473/merge'
        | where Branch != 'refs/pull/73057/merge'
        | where JobId != '20283731' // 7/29
        | where JobId != '20342414' // 8/6 ... 488x
        | where JobId != '20342402' // 8/6 ... 781x
        | where Branch != 'refs/pull/71405/merge'
        | where Branch != 'refs/pull/72869/merge'
        | where Branch != 'refs/pull/72814/merge'
        | where Branch != 'refs/pull/72886/merge'
        | where Branch != 'refs/pull/72882/merge'
        | where Branch != 'refs/pull/73055/merge'
        | where Branch != 'refs/pull/73200/merge'
        | where Branch != 'refs/pull/62863/merge'
        | where Branch != 'refs/pull/73061/merge'
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | where OS contains "ES"
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests(
    'System.Security.Cryptography', //testNameSubstring
    '', //methodName
    '',//ignored
    true,  //includePR
    true); //includePassedOnRerun

Known Issue Error Message

Fill the error message using known issues guidance.

{
    "ErrorPattern": ".*No hay.*extremos disponibles desde el asignador de extremos.*",
    "BuildRetry": false,
    "ExcludeConsoleLog": false
}

Report

Build Definition Test Pull Request
165740 dotnet/runtime System.Net.Http.Functional.Tests.PlatformHandler_HttpClientHandler_ServerCertificates_Http2_Test.UseCallback_SelfSignedCertificate_ExpectedPolicyErrors #81831
165735 dotnet/runtime System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.Manual_CertificateOnlySentWhenValid_Success #81728
164105 dotnet/runtime System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.Manual_CertificateOnlySentWhenValid_Success #78852
164104 dotnet/runtime System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_NetworkStream_Renegotiation_Succeeds #81822
163901 dotnet/runtime System.Formats.Cbor.Tests.CborReaderTests.CoseKeyHelpers_ECDsaParseCosePublicKey_HappyPath #80382

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 5

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions