[Regression] [Product bug] Test failure: System.Net.Sockets.Tests.TelemetryTest.EventSource_SocketConnectFailure_LogsConnectFailed(useDnsEndPoint: True)

[v-haren]

failed in job: runtime-libraries-coreclr outerloop 20201027.2

net6.0-Linux-Release-x64-CoreCLR_release-RedHat.7.Amd64.Open

Error message

Microsoft.DotNet.RemoteExecutor.RemoteExecutionException : Remote process failed with an unhandled exception.


Stack trace

Child exception:
  Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 1
Actual:   2
   at System.Net.Sockets.Tests.TelemetryTest.VerifyStartStopEvents(ConcurrentQueue`1 events, Boolean connect, Int32 expectedCount) in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/TelemetryTest.cs:line 396
   at System.Net.Sockets.Tests.TelemetryTest.VerifyConnectFailureEvents(ConcurrentQueue`1 events) in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/TelemetryTest.cs:line 324
   at System.Net.Sockets.Tests.TelemetryTest.<>c.<<EventSource_SocketConnectFailure_LogsConnectFailed>b__10_0>d.MoveNext() in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/TelemetryTest.cs:line 203
--- End of stack trace from previous location ---
   at Microsoft.DotNet.RemoteExecutor.Program.Main(String[] args) in /_/src/Microsoft.DotNet.RemoteExecutor/src/Program.cs:line 65

Child process:
  System.Net.Sockets.Tests, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 System.Net.Sockets.Tests.TelemetryTest+<>c System.Threading.Tasks.Task <EventSource_SocketConnectFailure_LogsConnectFailed>b__10_0(System.String, System.String)

Child arguments:
  Apm, True

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[v-haren]

failed again in job: runtime-libraries-coreclr outerloop 20201028.2

failed test: System.Net.Sockets.Tests.TelemetryTest.EventSource_ConnectAsyncCanceled_LogsConnectFailed(connectMethod: "Task", useDnsEndPoint: True)

net6.0-Linux-Release-x64-CoreCLR_release-RedHat.7.Amd64.Open

Error message

Microsoft.DotNet.RemoteExecutor.RemoteExecutionException : Remote process failed with an unhandled exception.


Stack trace

Child exception:
  Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 1
Actual:   2
   at System.Net.Sockets.Tests.TelemetryTest.VerifyStartStopEvents(ConcurrentQueue`1 events, Boolean connect, Int32 expectedCount) in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/TelemetryTest.cs:line 396
   at System.Net.Sockets.Tests.TelemetryTest.VerifyConnectFailureEvents(ConcurrentQueue`1 events) in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/TelemetryTest.cs:line 324
   at System.Net.Sockets.Tests.TelemetryTest.<>c.<<EventSource_ConnectAsyncCanceled_LogsConnectFailed>b__12_0>d.MoveNext() in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/TelemetryTest.cs:line 316
--- End of stack trace from previous location ---
   at Microsoft.DotNet.RemoteExecutor.Program.Main(String[] args) in /_/src/Microsoft.DotNet.RemoteExecutor/src/Program.cs:line 65

Child process:
  System.Net.Sockets.Tests, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 System.Net.Sockets.Tests.TelemetryTest+<>c System.Threading.Tasks.Task <EventSource_ConnectAsyncCanceled_LogsConnectFailed>b__12_0(System.String, System.String)

Child arguments:
  Task, True

[karelz]

Triage: Likely a regression - started to fail more often after change in the area recently.

[MihaZupan]

Kusto query I mentioned.

Initial failures on #43661

[stephentoub]

@MihaZupan, I've been unsuccessful in reproing this locally on either Windows or Ubuntu. Were you able to repro it?

[wfurt]

I see only 9 (2 debug, 9 release) failures in last 10 day. Only one failure with Task, 7 failures with APM.

Aside from the one above I see two other variants:

Child exception:
  Xunit.Sdk.ThrowsException: Assert.Throws() Failure
Expected: typeof(System.Exception)
Actual:   (No exception was thrown)
   at System.Net.Sockets.Tests.TelemetryTest.<>c__DisplayClass10_0.<<EventSource_SocketConnectFailure_LogsConnectFailed>b__2>d.MoveNext() in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/TelemetryTest.cs:line 185
--- End of stack trace from previous location ---
   at System.Diagnostics.Tracing.TestEventListener.RunWithCallbackAsync(Action`1 handler, Func`1 body) in /_/src/libraries/Common/tests/System/Diagnostics/Tracing/TestEventListener.cs:line 109
   at System.Net.Sockets.Tests.TelemetryTest.<>c.<<EventSource_SocketConnectFailure_LogsConnectFailed>b__10_0>d.MoveNext() in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/TelemetryTest.cs:line 179
--- End of stack trace from previous location ---
   at Microsoft.DotNet.RemoteExecutor.Program.Main(String[] args) in /_/src/Microsoft.DotNet.RemoteExecutor/src/Program.cs:line 65

Child process:
  System.Net.Sockets.Tests, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 System.Net.Sockets.Tests.TelemetryTest+<>c System.Threading.Tasks.Task <EventSource_SocketConnectFailure_LogsConnectFailed>b__10_0(System.String, System.String)

Child arguments:
  Sync, False

and

Child exception:
  System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000001, 11): Resource temporarily unavailable
   at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 385
   at System.Net.Dns.GetHostAddressesCore(String hostName, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 369
   at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__33_0(Object s) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 556
   at System.Threading.Tasks.Task`1.InnerInvoke() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:line 506
   at System.Threading.Tasks.Task.<>c.<.cctor>b__278_0(Object obj) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2359
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 275
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 300
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2321
--- End of stack trace from previous location ---
   at System.Net.Sockets.Tests.TelemetryTest.GetRemoteEndPointAsync(String useDnsEndPointString, Int32 port) in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/TelemetryTest.cs:line 70
   at System.Net.Sockets.Tests.TelemetryTest.<>c.<<EventSource_SocketConnectFailure_LogsConnectFailed>b__10_0>d.MoveNext() in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/TelemetryTest.cs:line 173
--- End of stack trace from previous location ---
   at Microsoft.DotNet.RemoteExecutor.Program.Main(String[] args) in /_/src/Microsoft.DotNet.RemoteExecutor/src/Program.cs:line 65

Child process:
  System.Net.Sockets.Tests, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 System.Net.Sockets.Tests.TelemetryTest+<>c System.Threading.Tasks.Task <EventSource_SocketConnectFailure_LogsConnectFailed>b__10_0(System.String, System.String)

Child arguments:
  Sync, False

[stephentoub]

Thanks. Yeah, I just ran 8000 iterations on both Windows and Linux over the course of a few hours and couldn't get it to happen. Obviously it can, of course, or it wouldn't be happening sporadically in CI.