TcpReceiveSendGetsCanceledByDispose tests fail on Fedora 38

[tmds]

The following tests fail on Fedora 38:

System.Net.Sockets.Tests.SendReceive_SpanSyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: False, owning: True)
System.Net.Sockets.Tests.SendReceive_SpanSyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: True, owning: True)
System.Net.Sockets.Tests.SendReceive_SpanSyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: True, dualModeClient: False, owning: True)
System.Net.Sockets.Tests.SendReceive_Sync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: False, owning: True)
System.Net.Sockets.Tests.SendReceive_Sync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: True, owning: True)
System.Net.Sockets.Tests.SendReceive_Sync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: True, dualModeClient: False, owning: True)
System.Net.Sockets.Tests.SendReceive_SyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: False, owning: True)
System.Net.Sockets.Tests.SendReceive_SyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: True, owning: True)
System.Net.Sockets.Tests.SendReceive_SyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: True, dualModeClient: False, owning: True)
System.Net.Sockets.Tests.SendReceive_SpanSync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: False, owning: True)
System.Net.Sockets.Tests.SendReceive_SpanSync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: True, owning: True)
System.Net.Sockets.Tests.SendReceive_SpanSync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: True, dualModeClient: False, owning: True)

The exceptions look like:

System.AggregateException: System.AggregateException : One or more errors occurred. (Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)) (Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)) (Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)) (Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)) (Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)) (Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)) (Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)) (Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null))\n---- Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)\n---- Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)\n---- Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)\n---- Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)\n---- Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)\n---- Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)\n---- Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)\n---- Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)
        at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /home/tester/runtime/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 102
at System.Net.Sockets.Tests.SendReceive`1.TcpReceiveSendGetsCanceledByDispose(Boolean receiveOrSend, Boolean ipv6Server, Boolean dualModeClient, Boolean owning) in /home/tester/runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 1032
--- End of stack trace from previous location ---
----- Inner Stack Trace #1 (Xunit.Sdk.EqualException) -----
at System.Net.Sockets.Tests.SendReceive`1.<>c__DisplayClass21_0.<<TcpReceiveSendGetsCanceledByDispose>b__0>d.MoveNext() in /home/tester/runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 1120
--- End of stack trace from previous location ---
at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /home/tester/runtime/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 94
----- Inner Stack Trace #2 (Xunit.Sdk.EqualException) -----
at System.Net.Sockets.Tests.SendReceive`1.<>c__DisplayClass21_0.<<TcpReceiveSendGetsCanceledByDispose>b__0>d.MoveNext() in /home/tester/runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 1120
--- End of stack trace from previous location ---
at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /home/tester/runtime/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 94
----- Inner Stack Trace #3 (Xunit.Sdk.EqualException) -----
at System.Net.Sockets.Tests.SendReceive`1.<>c__DisplayClass21_0.<<TcpReceiveSendGetsCanceledByDispose>b__0>d.MoveNext() in /home/tester/runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 1120
--- End of stack trace from previous location ---
at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /home/tester/runtime/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 94
----- Inner Stack Trace #4 (Xunit.Sdk.EqualException) -----
at System.Net.Sockets.Tests.SendReceive`1.<>c__DisplayClass21_0.<<TcpReceiveSendGetsCanceledByDispose>b__0>d.MoveNext() in /home/tester/runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 1120
--- End of stack trace from previous location ---
at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /home/tester/runtime/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 94
----- Inner Stack Trace #5 (Xunit.Sdk.EqualException) -----
at System.Net.Sockets.Tests.SendReceive`1.<>c__DisplayClass21_0.<<TcpReceiveSendGetsCanceledByDispose>b__0>d.MoveNext() in /home/tester/runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 1120
--- End of stack trace from previous location ---
at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /home/tester/runtime/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 94
----- Inner Stack Trace #6 (Xunit.Sdk.EqualException) -----
at System.Net.Sockets.Tests.SendReceive`1.<>c__DisplayClass21_0.<<TcpReceiveSendGetsCanceledByDispose>b__0>d.MoveNext() in /home/tester/runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 1120
--- End of stack trace from previous location ---
at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /home/tester/runtime/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 94
----- Inner Stack Trace #7 (Xunit.Sdk.EqualException) -----
at System.Net.Sockets.Tests.SendReceive`1.<>c__DisplayClass21_0.<<TcpReceiveSendGetsCanceledByDispose>b__0>d.MoveNext() in /home/tester/runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 1120
--- End of stack trace from previous location ---
at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /home/tester/runtime/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 94
----- Inner Stack Trace #8 (Xunit.Sdk.EqualException) -----
at System.Net.Sockets.Tests.SendReceive`1.<>c__DisplayClass21_0.<<TcpReceiveSendGetsCanceledByDispose>b__0>d.MoveNext() in /home/tester/runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 1120
--- End of stack trace from previous location ---
at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /home/tester/runtime/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 94

These tests don't fail on Fedora 37, so the behavior is possibly triggered by a change in kernel behavior.

I will investigate further when I have some time.

cc @omajid

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

The following tests fail on Fedora 38:

System.Net.Sockets.Tests.SendReceive_SpanSyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: False, owning: True)
System.Net.Sockets.Tests.SendReceive_SpanSyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: True, owning: True)
System.Net.Sockets.Tests.SendReceive_SpanSyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: True, dualModeClient: False, owning: True)
System.Net.Sockets.Tests.SendReceive_Sync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: False, owning: True)
System.Net.Sockets.Tests.SendReceive_Sync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: True, owning: True)
System.Net.Sockets.Tests.SendReceive_Sync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: True, dualModeClient: False, owning: True)
System.Net.Sockets.Tests.SendReceive_SyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: False, owning: True)
System.Net.Sockets.Tests.SendReceive_SyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: True, owning: True)
System.Net.Sockets.Tests.SendReceive_SyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: True, dualModeClient: False, owning: True)
System.Net.Sockets.Tests.SendReceive_SpanSync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: False, owning: True)
System.Net.Sockets.Tests.SendReceive_SpanSync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: True, owning: True)
System.Net.Sockets.Tests.SendReceive_SpanSync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: True, dualModeClient: False, owning: True)

The exceptions look like:

System.AggregateException: System.AggregateException : One or more errors occurred. (Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)) (Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)) (Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)) (Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)) (Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)) (Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)) (Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)) (Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null))\n---- Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)\n---- Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)\n---- Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)\n---- Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)\n---- Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)\n---- Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)\n---- Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)\n---- Assert.Equal() Failure\nExpected: ConnectionReset\nActual:   (null)
        at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /home/tester/runtime/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 102
at System.Net.Sockets.Tests.SendReceive`1.TcpReceiveSendGetsCanceledByDispose(Boolean receiveOrSend, Boolean ipv6Server, Boolean dualModeClient, Boolean owning) in /home/tester/runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 1032
--- End of stack trace from previous location ---
----- Inner Stack Trace #1 (Xunit.Sdk.EqualException) -----
at System.Net.Sockets.Tests.SendReceive`1.<>c__DisplayClass21_0.<<TcpReceiveSendGetsCanceledByDispose>b__0>d.MoveNext() in /home/tester/runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 1120
--- End of stack trace from previous location ---
at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /home/tester/runtime/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 94
----- Inner Stack Trace #2 (Xunit.Sdk.EqualException) -----
at System.Net.Sockets.Tests.SendReceive`1.<>c__DisplayClass21_0.<<TcpReceiveSendGetsCanceledByDispose>b__0>d.MoveNext() in /home/tester/runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 1120
--- End of stack trace from previous location ---
at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /home/tester/runtime/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 94
----- Inner Stack Trace #3 (Xunit.Sdk.EqualException) -----
at System.Net.Sockets.Tests.SendReceive`1.<>c__DisplayClass21_0.<<TcpReceiveSendGetsCanceledByDispose>b__0>d.MoveNext() in /home/tester/runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 1120
--- End of stack trace from previous location ---
at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /home/tester/runtime/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 94
----- Inner Stack Trace #4 (Xunit.Sdk.EqualException) -----
at System.Net.Sockets.Tests.SendReceive`1.<>c__DisplayClass21_0.<<TcpReceiveSendGetsCanceledByDispose>b__0>d.MoveNext() in /home/tester/runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 1120
--- End of stack trace from previous location ---
at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /home/tester/runtime/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 94
----- Inner Stack Trace #5 (Xunit.Sdk.EqualException) -----
at System.Net.Sockets.Tests.SendReceive`1.<>c__DisplayClass21_0.<<TcpReceiveSendGetsCanceledByDispose>b__0>d.MoveNext() in /home/tester/runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 1120
--- End of stack trace from previous location ---
at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /home/tester/runtime/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 94
----- Inner Stack Trace #6 (Xunit.Sdk.EqualException) -----
at System.Net.Sockets.Tests.SendReceive`1.<>c__DisplayClass21_0.<<TcpReceiveSendGetsCanceledByDispose>b__0>d.MoveNext() in /home/tester/runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 1120
--- End of stack trace from previous location ---
at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /home/tester/runtime/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 94
----- Inner Stack Trace #7 (Xunit.Sdk.EqualException) -----
at System.Net.Sockets.Tests.SendReceive`1.<>c__DisplayClass21_0.<<TcpReceiveSendGetsCanceledByDispose>b__0>d.MoveNext() in /home/tester/runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 1120
--- End of stack trace from previous location ---
at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /home/tester/runtime/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 94
----- Inner Stack Trace #8 (Xunit.Sdk.EqualException) -----
at System.Net.Sockets.Tests.SendReceive`1.<>c__DisplayClass21_0.<<TcpReceiveSendGetsCanceledByDispose>b__0>d.MoveNext() in /home/tester/runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 1120
--- End of stack trace from previous location ---
at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /home/tester/runtime/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 94

These tests don't fail on Fedora 37, so the behavior is possibly triggered by a change in kernel behavior.

I will investigate further when I have some time.

cc @omajid

Author:	tmds
Assignees:	-
Labels:	area-System.Net.Sockets
Milestone:	-

[karelz]

@tmds please do not remove untriaged label. We do it only when we set milestone. Otherwise issues might slip through :(

[tmds]

@tmds please do not remove untriaged label. We do it only when we set milestone. Otherwise issues might slip through :(

Ok, I haven't had this feedback before. I'll stop removing the label.

[directhex]

@tmds this is failing for you on x64 with Core, right?

These exact failures are plaguing our partners at IBM, for their configurations (Mono on PPC64 little endian, Mono on s390x). At the very least that seems to imply it's not a runtime-specific issue.

e.g. https://dev.azure.com/dnceng-public/public/_build/results?buildId=421005&view=results

[tmds]

These exact failures are plaguing our partners at IBM, for their configurations (Mono on PPC64 little endian, Mono on s390x). At the very least that seems to imply it's not a runtime-specific issue.

Right. As mentioned in the initial comment, I think it's in the kernel.

Note that these tests don't fail on our CI when they run on ppc64le with RHEL 8.
And, RHEL8 has a different kernel than what CI is using here.