Test failure:System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_DangerousAcceptAllCertificatesValidator_Test.SetDelegate_ConnectionSucceeds(acceptedProtocol: Tls | Tls11 | Tls12, requestOnlyThisProtocol: False)

[VincentBu]

Failures 6/13-8/13 (incl. PRs):

Day	Run	OS	Notes
6/23-6/24	PR #54618	Windows.10.Amd64.Android.Open	18x failures -- specific to the PR, unrelated
8/2	Official run	OSX.1013.Amd64.Open	This reported failure
8/11	PR #57217	misc	32x failures - unrelated, specific to the PR

Run: runtime 20210802.77

Failed test:

net6.0-OSX-Release-x64-CoreCLR_release-OSX.1013.Amd64.Open
System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_DangerousAcceptAllCertificatesValidator_Test.SetDelegate_ConnectionSucceeds(acceptedProtocol: Tls | Tls11 | Tls12, requestOnlyThisProtocol: False)

Error message:

System.AggregateException : One or more errors occurred. (One or more errors occurred. ( Received an unexpected EOF or 0 bytes from the transport stream.)) (One or more errors occurred. (Socket is already connected [::ffff:127.0.0.1]:49713 (127.0.0.1:49713)))
---- System.AggregateException : One or more errors occurred. ( Received an unexpected EOF or 0 bytes from the transport stream.)
-------- System.IO.IOException :  Received an unexpected EOF or 0 bytes from the transport stream.
---- System.AggregateException : One or more errors occurred. (Socket is already connected [::ffff:127.0.0.1]:49713 (127.0.0.1:49713))
-------- System.Net.Http.HttpRequestException : Socket is already connected [::ffff:127.0.0.1]:49713 (127.0.0.1:49713)
------------ System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Socket is already connected [::ffff:127.0.0.1]:49713


Stack trace
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 89
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 55
   at System.Net.Http.Functional.Tests.HttpClientHandler_DangerousAcceptAllCertificatesValidator_Test.<>c__DisplayClass4_0.<<SetDelegate_ConnectionSucceeds>b__0>d.MoveNext() in /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.AcceptAllCerts.cs:line 78
--- End of stack trace from previous location ---
   at System.Net.Test.Common.LoopbackServer.CreateServerAsync(Func`2 funcAsync, Options options) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 103
   at System.Net.Http.Functional.Tests.HttpClientHandler_DangerousAcceptAllCertificatesValidator_Test.SetDelegate_ConnectionSucceeds(SslProtocols acceptedProtocol, Boolean requestOnlyThisProtocol) in /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.AcceptAllCerts.cs:line 76
--- End of stack trace from previous location ---
----- Inner Stack Trace #1 (System.AggregateException) -----

----- Inner Stack Trace -----
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 464
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 380
   at System.Net.Test.Common.LoopbackServer.Connection.CreateAsync(SocketWrapper socket, Stream stream, Options httpOptions) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 489
   at System.Net.Test.Common.LoopbackServer.EstablishConnectionAsync() in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 155
   at System.Net.Test.Common.LoopbackServer.AcceptConnectionAsync(Func`2 funcAsync) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 163
   at System.Net.Test.Common.LoopbackServer.AcceptConnectionSendResponseAndCloseAsync(HttpStatusCode statusCode, String additionalHeaders, String content) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 205
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 72
----- Inner Stack Trace #2 (System.AggregateException) -----

----- Inner Stack Trace -----
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1462
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1406
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1469
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 480
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken) in /_/src/libraries/Common/src/System/Threading/Tasks/TaskCompletionSourceWithCancellation.cs:line 23
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 558
   at System.Net.Http.HttpConnectionPool.SendUsingHttp11Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 981
   at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1015
   at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1028
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1038
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 553
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 64
----- Inner Stack Trace -----
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 3187
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 849
   at System.Net.Sockets.Socket.Connect(String host, Int32 port) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 902
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 849
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1462

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run: runtime 20210802.77

Failed test:

net6.0-OSX-Release-x64-CoreCLR_release-OSX.1013.Amd64.Open
System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_DangerousAcceptAllCertificatesValidator_Test.SetDelegate_ConnectionSucceeds(acceptedProtocol: Tls | Tls11 | Tls12, requestOnlyThisProtocol: False)

Error message:

System.AggregateException : One or more errors occurred. (One or more errors occurred. ( Received an unexpected EOF or 0 bytes from the transport stream.)) (One or more errors occurred. (Socket is already connected [::ffff:127.0.0.1]:49713 (127.0.0.1:49713)))
---- System.AggregateException : One or more errors occurred. ( Received an unexpected EOF or 0 bytes from the transport stream.)
-------- System.IO.IOException :  Received an unexpected EOF or 0 bytes from the transport stream.
---- System.AggregateException : One or more errors occurred. (Socket is already connected [::ffff:127.0.0.1]:49713 (127.0.0.1:49713))
-------- System.Net.Http.HttpRequestException : Socket is already connected [::ffff:127.0.0.1]:49713 (127.0.0.1:49713)
------------ System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Socket is already connected [::ffff:127.0.0.1]:49713


Stack trace
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 89
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 55
   at System.Net.Http.Functional.Tests.HttpClientHandler_DangerousAcceptAllCertificatesValidator_Test.<>c__DisplayClass4_0.<<SetDelegate_ConnectionSucceeds>b__0>d.MoveNext() in /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.AcceptAllCerts.cs:line 78
--- End of stack trace from previous location ---
   at System.Net.Test.Common.LoopbackServer.CreateServerAsync(Func`2 funcAsync, Options options) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 103
   at System.Net.Http.Functional.Tests.HttpClientHandler_DangerousAcceptAllCertificatesValidator_Test.SetDelegate_ConnectionSucceeds(SslProtocols acceptedProtocol, Boolean requestOnlyThisProtocol) in /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.AcceptAllCerts.cs:line 76
--- End of stack trace from previous location ---
----- Inner Stack Trace #1 (System.AggregateException) -----

----- Inner Stack Trace -----
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 464
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 380
   at System.Net.Test.Common.LoopbackServer.Connection.CreateAsync(SocketWrapper socket, Stream stream, Options httpOptions) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 489
   at System.Net.Test.Common.LoopbackServer.EstablishConnectionAsync() in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 155
   at System.Net.Test.Common.LoopbackServer.AcceptConnectionAsync(Func`2 funcAsync) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 163
   at System.Net.Test.Common.LoopbackServer.AcceptConnectionSendResponseAndCloseAsync(HttpStatusCode statusCode, String additionalHeaders, String content) in /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs:line 205
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 72
----- Inner Stack Trace #2 (System.AggregateException) -----

----- Inner Stack Trace -----
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1462
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1406
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1469
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 480
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken) in /_/src/libraries/Common/src/System/Threading/Tasks/TaskCompletionSourceWithCancellation.cs:line 23
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 558
   at System.Net.Http.HttpConnectionPool.SendUsingHttp11Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 981
   at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1015
   at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1028
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1038
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 553
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 64
----- Inner Stack Trace -----
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 3187
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 849
   at System.Net.Sockets.Socket.Connect(String host, Int32 port) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 902
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 849
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1462

Author:	VincentBu
Assignees:	-
Labels:	arch-x64, area-System.Net.Http
Milestone:	-

[wfurt]

I'm not sure if this matters to underlying Socket issue but briefly looking at the code I was curious why SendAsync would cause synchronous IO.

It seems like the culprit is

runtime/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs

Line 464 in a0ff4e1

connection = await CreateHttp11ConnectionAsync(request, false, cts.Token).ConfigureAwait(false);

from #53851

This may contribute to #56477
cc: @geoffkizer @stephentoub

BTW my plan is to add Assert to debug build for this particular failure on macOS so we can get core.
It seems like the 'Socket is already connected ' on macOS is somewhat prevalent recently.

[ManickaP]

We suspect this is the same problem as #56477 which is in PR.

[ManickaP]

Triage: punting to future, might be fixed by #56966

[karelz]

Closing as addressed.
We can reopen in Future if it happens again.