ClientAndServer_OneOrBothUseDefault_Ok failed in CI with ObjectDisposedException

[ViktorHofer]

Tests:

• System.Net.Security.Tests.ApmSslStreamSystemDefaultTest.ClientAndServer_OneOrBothUseDefault_Ok
• System.Net.Security.Tests.SyncSslStreamSystemDefaultTest.ClientAndServer_OneOrBothUseDefault_Ok

Hits:

• dotnet/corefx#41936 - System.Net.Security.Tests.AsyncSslStreamSystemDefaultTest.ClientAndServer_OneOrBothUseDefault_Ok(clientProtocols: Tls | Tls11 | Tls12, serverProtocols: null) failing in CI with ObjectDisposedException

https://dnceng.visualstudio.com/public/_build/results?buildId=331181&view=ms.vss-test-web.build-test-results-tab&runId=9703666&resultId=167383&paneView=debug

Configuration: netcoreapp-Windows_NT-Release-x64-Windows.81.Amd64.Open

System.Net.Security.Tests.ApmSslStreamSystemDefaultTest.ClientAndServer_OneOrBothUseDefault_Ok(clientProtocols: Tls11, serverProtocols: Ssl2 | Default | Tls11 | Tls12)

System.ObjectDisposedException : Safe handle has been closed.\r\nObject name: 'SafeHandle'.

at System.Runtime.InteropServices.SafeHandle.DangerousAddRef(Boolean& success) in /_/src/System.Private.CoreLib/shared/System/Runtime/InteropServices/SafeHandle.cs:line 158
   at System.Net.Security.SafeCredentialReference..ctor(SafeFreeCredentials target) in /_/src/Common/src/Interop/Windows/SspiCli/SecuritySafeHandles.cs:line 354
   at System.Net.Security.SafeCredentialReference.CreateReference(SafeFreeCredentials target) in /_/src/Common/src/Interop/Windows/SspiCli/SecuritySafeHandles.cs:line 340
   at System.Net.Security.SslSessionsCache.CacheCredential(SafeFreeCredentials creds, Byte[] thumbPrint, SslProtocols sslProtocols, Boolean isServer, EncryptionPolicy encryptionPolicy) in /_/src/System.Net.Security/src/System/Net/Security/SslSessionsCache.cs:line 201
   at System.Net.Security.SecureChannel.GenerateToken(Byte[] input, Int32 offset, Int32 count, Byte[]& output) in /_/src/System.Net.Security/src/System/Net/Security/SecureChannel.cs:line 846
   at System.Net.Security.SecureChannel.NextMessage(Byte[] incoming, Int32 offset, Int32 count) in /_/src/System.Net.Security/src/System/Net/Security/SecureChannel.cs:line 738
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) in /_/src/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 446
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest) in /_/src/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 662
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest) in /_/src/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 617
   at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest) in /_/src/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 796
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Security.SslStream.<ThrowIfExceptional>g__ThrowExceptional|141_0(ExceptionDispatchInfo e) in /_/src/System.Net.Security/src/System/Net/Security/SslStream.cs:line 844
   at System.Net.Security.SslStream.ThrowIfExceptional() in /_/src/System.Net.Security/src/System/Net/Security/SslStream.cs:line 845
   at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult) in /_/src/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 439
   at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result) in /_/src/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 410
   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult) in /_/src/System.Net.Security/src/System/Net/Security/SslStream.cs:line 241
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/FutureFactory.cs:line 514
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 83
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 111
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /_/src/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 71
   at System.Net.Security.Tests.SslStreamSystemDefaultTest.ClientAndServer_OneOrBothUseDefault_Ok(Nullable`1 clientProtocols, Nullable`1 serverProtocols) in /_/src/System.Net.Security/tests/FunctionalTests/SslStreamSystemDefaultsTest.cs:line 63
--- End of stack trace from previous location where exception was thrown ---

cc @davidsh, @wfurt, @scalablecory, @eiriktsarpalis, @karelz

[ViktorHofer]

Happened again: https://dnceng.visualstudio.com/public/_build/results?buildId=395007&view=ms.vss-test-web.build-test-results-tab&runId=12313218&resultId=165435&paneView=debug

[karelz]

Triage: We should check why it regressed recently
@wfurt please check if it reproduced in the past and if it will be impacted by your SSL changes.

[jkotas]

Hit in #33664

Test name: System.Net.Security.Tests.ApmSslStreamSystemDefaultTest.ClientAndServer_OneOrBothUseDefault_Ok(clientProtocols: Default, serverProtocols: Ssl2 | Default | Tls11 | Tls12)

Log :

    System.Net.Security.Tests.ApmSslStreamSystemDefaultTest.ClientAndServer_OneOrBothUseDefault_Ok(clientProtocols: Default, serverProtocols: Ssl2 | Default | Tls11 | Tls12) [FAIL]
      System.ObjectDisposedException : Safe handle has been closed.
      Object name: 'SafeHandle'.
      Stack Trace:
        /_/src/libraries/System.Private.CoreLib/src/System/Runtime/InteropServices/SafeHandle.cs(149,0): at System.Runtime.InteropServices.SafeHandle.DangerousAddRef(Boolean& success)
        /_/src/libraries/Common/src/Interop/Windows/SspiCli/SecuritySafeHandles.cs(354,0): at System.Net.Security.SafeCredentialReference..ctor(SafeFreeCredentials target)
        /_/src/libraries/Common/src/Interop/Windows/SspiCli/SecuritySafeHandles.cs(341,0): at System.Net.Security.SafeCredentialReference.CreateReference(SafeFreeCredentials target)
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SslSessionsCache.cs(201,0): at System.Net.Security.SslSessionsCache.CacheCredential(SafeFreeCredentials creds, Byte[] thumbPrint, SslProtocols sslProtocols, Boolean isServer, EncryptionPolicy encryptionPolicy)
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SecureChannel.cs(830,0): at System.Net.Security.SecureChannel.GenerateToken(ReadOnlySpan`1 inputBuffer, Byte[]& output)
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SecureChannel.cs(734,0): at System.Net.Security.SecureChannel.NextMessage(ReadOnlySpan`1 incomingBuffer)
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs(375,0): at System.Net.Security.SslStream.ProcessBlob(Int32 frameSize)
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs(342,0): at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs(268,0): at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
        /_/src/libraries/Common/src/System/Threading/Tasks/TaskToApm.cs(42,0): at System.Threading.Tasks.TaskToApm.End(IAsyncResult asyncResult)
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs(233,0): at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/FutureFactory.cs(513,0): at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(83,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(111,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(71,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout)
        /_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSystemDefaultsTest.cs(60,0): at System.Net.Security.Tests.SslStreamSystemDefaultTest.ClientAndServer_OneOrBothUseDefault_Ok(Nullable`1 clientProtocols, Nullable`1 serverProtocols)
        --- End of stack trace from previous location ---
    System.Net.Security.Tests.SyncSslStreamSystemDefaultTest.ClientAndServer_OneOrBothUseDefault_Ok(clientProtocols: Ssl2 | Default | Tls11 | Tls12, serverProtocols: Default) [FAIL]
      System.NullReferenceException : Object reference not set to an instance of an object.
      Stack Trace:
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SslSessionsCache.cs(199,0): at System.Net.Security.SslSessionsCache.CacheCredential(SafeFreeCredentials creds, Byte[] thumbPrint, SslProtocols sslProtocols, Boolean isServer, EncryptionPolicy encryptionPolicy)
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SecureChannel.cs(830,0): at System.Net.Security.SecureChannel.GenerateToken(ReadOnlySpan`1 inputBuffer, Byte[]& output)
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SecureChannel.cs(726,0): at System.Net.Security.SecureChannel.NextMessage(ReadOnlySpan`1 incomingBuffer)
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs(375,0): at System.Net.Security.SslStream.ProcessBlob(Int32 frameSize)
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs(342,0): at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs(268,0): at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs(208,0): at System.Net.Security.SslStream.ProcessAuthentication(Boolean isAsync, Boolean isApm, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs(345,0): at System.Net.Security.SslStream.AuthenticateAsServer(SslServerAuthenticationOptions sslServerAuthenticationOptions)
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs(337,0): at System.Net.Security.SslStream.AuthenticateAsServer(X509Certificate serverCertificate, Boolean clientCertificateRequired, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
        /_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSystemDefaultsTest.cs(164,0): at System.Net.Security.Tests.SyncSslStreamSystemDefaultTest.<>c__DisplayClass1_0.<AuthenticateServerAsync>b__0()
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs(2364,0): at System.Threading.Tasks.Task.InnerInvoke()
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs(2350,0): at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj)
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs(289,0): at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
        --- End of stack trace from previous location ---
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs(314,0): at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
        /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs(2314,0): at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(83,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(111,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(71,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout)
        /_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSystemDefaultsTest.cs(60,0): at System.Net.Security.Tests.SslStreamSystemDefaultTest.ClientAndServer_OneOrBothUseDefault_Ok(Nullable`1 clientProtocols, Nullable`1 serverProtocols)
        --- End of stack trace from previous location ---
  Finished:    System.Net.Security.Tests

[v-haren]

failed again in job: runtime 20201027.54

failed test: System.Net.Security.Tests.ApmSslStreamSystemDefaultTest.ClientAndServer_OneOrBothUseDefault_Ok(clientProtocols: Tls | Tls11 | Tls12, serverProtocols: Default)

net6.0-Windows_NT-Release-x64-CoreCLR_release-Windows.81.Amd64.Open

Error message

System.ObjectDisposedException : Safe handle has been closed.
Object name: 'SafeHandle'.


Stack trace
   at System.Runtime.InteropServices.SafeHandle.DangerousAddRef(Boolean& success) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/InteropServices/SafeHandle.cs:line 148
   at System.Net.Security.SafeCredentialReference..ctor(SafeFreeCredentials target) in /_/src/libraries/Common/src/Interop/Windows/SspiCli/SecuritySafeHandles.cs:line 353
   at System.Net.Security.SafeCredentialReference.CreateReference(SafeFreeCredentials target) in /_/src/libraries/Common/src/Interop/Windows/SspiCli/SecuritySafeHandles.cs:line 340
   at System.Net.Security.SslSessionsCache.CacheCredential(SafeFreeCredentials creds, Byte[] thumbPrint, SslProtocols sslProtocols, Boolean isServer, EncryptionPolicy encryptionPolicy) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslSessionsCache.cs:line 197
   at System.Net.Security.SecureChannel.GenerateToken(ReadOnlySpan`1 inputBuffer, Byte[]& output) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SecureChannel.cs:line 819
   at System.Net.Security.SecureChannel.NextMessage(ReadOnlySpan`1 incomingBuffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SecureChannel.cs:line 727
   at System.Net.Security.SslStream.ProcessBlob(Int32 frameSize) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 577
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 539
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 356
   at System.Threading.Tasks.TaskToApm.End(IAsyncResult asyncResult) in /_/src/libraries/Common/src/System/Threading/Tasks/TaskToApm.cs:line 42
   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 212
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/FutureFactory.cs:line 512
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 96
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 124
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 84
   at System.Net.Security.Tests.SslStreamSystemDefaultTest.ClientAndServer_OneOrBothUseDefault_Ok(Nullable`1 clientProtocols, Nullable`1 serverProtocols) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSystemDefaultsTest.cs:line 85
--- End of stack trace from previous location ---