Assert while running System.Net.Security tests

[wfurt]

I did MANY runs (Ubuntu 18.04) and this is first occurrence so it may be hard to reproduce. The assert is guarded by #if DEBUG so it should impact released product but it suggests that there was prior error we did not handle or there is some race condition with other thread.

Process terminated. Assertion failed.
OpenSsl error queue is not empty, run: 'openssl errstr 1409442E' for original error.
   at Interop.OpenSsl.Encrypt(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& output, SslErrorCode& errorCode) in /home/furt/github/wfurt-runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 301
   at System.Net.Security.SslStreamPal.EncryptDecryptHelper(SafeDeleteContext securityContext, ReadOnlyMemory`1 input, Int32 offset, Int32 size, Boolean encrypt, Byte[]& output, Int32& resultSize) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 163
   at System.Net.Security.SslStreamPal.EncryptMessage(SafeDeleteContext securityContext, ReadOnlyMemory`1 input, Int32 headerSize, Int32 trailerSize, Byte[]& output, Int32& resultSize) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 47
   at System.Net.Security.SecureChannel.Encrypt(ReadOnlyMemory`1 buffer, Byte[]& output, Int32& resultSize) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Security/src/System/Net/Security/SecureChannel.cs:line 873
   at System.Net.Security.SslStream.EncryptData(ReadOnlyMemory`1 buffer, Byte[]& outBuffer, Int32& outSize) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 180
   at System.Net.Security.SslStream.WriteSingleChunk[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 644
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 986
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 42
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 769
   at System.IO.Stream.WriteByte(Byte value) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 692
   at System.IO.Tests.WrappingConnectedStreamConformanceTests.<>c__DisplayClass9_1.<NestedWithinSelf_ReadWrite_Success>b__0() in /home/furt/github/wfurt-runtime/src/libraries/Common/tests/Tests/System/IO/StreamConformanceTests.cs:line 2757
   at System.Threading.Tasks.Task.InnerInvoke() in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2371
   at System.Threading.Tasks.Task.<>c.<.cctor>b__278_0(Object obj) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2359
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 275
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2321
   at System.Threading.Tasks.Task.ExecuteEntryUnsafe(Thread threadPoolThread) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2255
   at System.Threading.Tasks.Task.ExecuteFromThreadPool(Thread threadPoolThread) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2246
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.cs:line 705
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 54
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state) in /home/furt/github/wfurt-runtime/src/coreclr/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 43
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 186
   at System.Threading.ThreadHelper.ThreadStart() in /home/furt/github/wfurt-runtime/src/coreclr/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 92

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Description:	I did MANY runs (Ubuntu 18.04) and this is first occurrence so it may be hard to reproduce. The assert is guarded by #if DEBUG so it should impact released product but it suggests that there was prior error we did not handle or there is some race condition with other thread. Process terminated. Assertion failed. OpenSsl error queue is not empty, run: 'openssl errstr 1409442E' for original error. at Interop.OpenSsl.Encrypt(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& output, SslErrorCode& errorCode) in /home/furt/github/wfurt-runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 301 at System.Net.Security.SslStreamPal.EncryptDecryptHelper(SafeDeleteContext securityContext, ReadOnlyMemory`1 input, Int32 offset, Int32 size, Boolean encrypt, Byte[]& output, Int32& resultSize) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 163 at System.Net.Security.SslStreamPal.EncryptMessage(SafeDeleteContext securityContext, ReadOnlyMemory`1 input, Int32 headerSize, Int32 trailerSize, Byte[]& output, Int32& resultSize) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 47 at System.Net.Security.SecureChannel.Encrypt(ReadOnlyMemory`1 buffer, Byte[]& output, Int32& resultSize) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Security/src/System/Net/Security/SecureChannel.cs:line 873 at System.Net.Security.SslStream.EncryptData(ReadOnlyMemory`1 buffer, Byte[]& outBuffer, Int32& outSize) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 180 at System.Net.Security.SslStream.WriteSingleChunk[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 644 at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 986 at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 42 at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 769 at System.IO.Stream.WriteByte(Byte value) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 692 at System.IO.Tests.WrappingConnectedStreamConformanceTests.<>c__DisplayClass9_1.<NestedWithinSelf_ReadWrite_Success>b__0() in /home/furt/github/wfurt-runtime/src/libraries/Common/tests/Tests/System/IO/StreamConformanceTests.cs:line 2757 at System.Threading.Tasks.Task.InnerInvoke() in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2371 at System.Threading.Tasks.Task.<>c.<.cctor>b__278_0(Object obj) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2359 at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 275 at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2321 at System.Threading.Tasks.Task.ExecuteEntryUnsafe(Thread threadPoolThread) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2255 at System.Threading.Tasks.Task.ExecuteFromThreadPool(Thread threadPoolThread) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2246 at System.Threading.ThreadPoolWorkQueue.Dispatch() in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.cs:line 705 at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 54 at System.Threading.ThreadHelper.ThreadStart_Context(Object state) in /home/furt/github/wfurt-runtime/src/coreclr/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 43 at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 186 at System.Threading.ThreadHelper.ThreadStart() in /home/furt/github/wfurt-runtime/src/coreclr/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 92
Author:	wfurt
Assignees:	-
Labels:	area-System.Net.Security, os-linux
Milestone:	-

[karelz]

Triage: Add code to dump the content of the error when we hit this -- then see if we can get more useful info

[akoeplinger]

Just hit this in https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-53808-merge-00beadfa06ad47ee82/System.Net.Security.Tests/console.e87a3360.log?sv=2019-07-07&se=2021-07-01T09%3A29%3A53Z&sr=c&sp=rl&sig=TJKWwOo70AYRz5SicBpXmxFtD3nCVhvJixxPDXwWLBs%3D

OpenSsl error queue is not empty, run: 'openssl errstr 1409442E' for original error.
   at Interop.OpenSsl.Decrypt(SafeSslHandle context, Span`1 buffer, SslErrorCode& errorCode) in /_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 355
   at System.Net.Security.SslStreamPal.DecryptMessage(SafeDeleteSslContext securityContext, Byte[] buffer, Int32& offset, Int32& count) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 69
   at System.Net.Security.SecureChannel.Decrypt(Byte[] payload, Int32& offset, Int32& count) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SecureChannel.cs:line 920
   at System.Net.Security.SslStream.PrivateDecryptData(Byte[] buffer, Int32& offset, Int32& count) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 194
   at System.Net.Security.SslStream.DecryptData() in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 189
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer, Boolean renegotiation) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 902
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 183
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 324
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 795
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3368
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:line 400
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 446
   at System.IO.StreamBuffer.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/Common/src/System/Net/StreamBuffer.cs:line 264
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 268
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 328
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 708
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 63
   at System.Threading.Thread.StartCallback() in /_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 105

The error message is:

$ openssl errstr 1409442E
error:1409442E:SSL routines:func(148):tlsv1 alert protocol version

Seems to be an issue with some TLSv1 case.