Skip to content

MsQuic tests hang / long running #56487

New issue
New issue
Closed
Closed
MsQuic tests hang / long running#56487
Labels
area-System.Net.Quictest-run-coreTest failures in .NET Core test runs
Milestone
 
6.0.0
@karelz

Description

@karelz
karelz
opened on Jul 28, 2021

Affected tests:

  • System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests:
    • ReadWrite_Success
    • ReadWrite_MessagesSmallerThanReadBuffer_Success
  • System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider:
    • ReadWrite_Random_Success
    • LargeDataSentAndReceived
    • MultipleReadsAndWrites

Failures 3/29-7/28 (incl. PRs):

  • Always at least 2 tests -- mostly 2 tests hang at the same time
  • Most hangs are on Mono
  • Looks like a regression on 7/13
Day Run Runtime Test 1 (MsQuicQuicStreamConformanceTests) Test 2 (QuicStreamTests_MsQuicProvider) Additional tests
7/13 PR #55550 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success 3rd: sQuicTests.ByteMixingOrNativeAVE_MinimalFailingTest
7/13 PR #55332 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success 3rd: MsQuicTests.ByteMixingOrNativeAVE_MinimalFailingTest
7/14 PR #55599 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success 3rd: MsQuicTests.ByteMixingOrNativeAVE_MinimalFailingTest
7/14 PR #54293 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #55533 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/14 PR #55533 CoreCLR ReadWrite_Success LargeDataSentAndReceived --
7/14 PR #55594 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 Official run Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #55613 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #55636 Mono ReadWrite_Success LargeDataSentAndReceived --
7/14 PR #55648 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #54453 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #55667 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #55596 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #54640 Mono ReadWrite_Success ReadWrite_Random_Success --
7/15 PR #52244 Mono ReadWrite_Success ReadWrite_Random_Success --
7/15 Official run Mono ReadWrite_Success MultipleReadsAndWrites --
7/15 PR #55719 Mono ReadWrite_Success ReadWrite_Random_Success --
7/15 PR #53505 Mono ReadWrite_Success ReadWrite_Random_Success --
7/15 PR #55658 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/15 PR #55710 Mono ReadWrite_Success ReadWrite_Random_Success --
7/15 PR #52699 Mono ReadWrite_Success MultipleReadsAndWrites --
7/16 PR #55776 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/16 PR #55772 Mono ReadWrite_Success MultipleReadsAndWrites --
7/16 PR #55724 Mono ReadWrite_Success ReadWrite_Random_Success --
7/16 PR #54640 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/16 PR #55801 Mono ReadWrite_Success ReadWrite_Random_Success --
7/16 PR #55787 Mono ReadWrite_Success ReadWrite_Random_Success --
7/16 PR #55769 Mono ReadWrite_Success ReadWrite_Random_Success --
7/16 Official run CoreCLR ReadWrite_Success LargeDataSentAndReceived --
7/16 PR #55797 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/17 PR #55875 Mono ReadWrite_Success ReadWrite_Random_Success --
7/17 PR #55875 Mono ReadWrite_Success ReadWrite_Random_Success --
7/18 Official run Mono ReadWrite_Success LargeDataSentAndReceived --
7/18 PR #55877 Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 PR #54640 Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 PR #55912 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/19 Official run Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 PR #55883 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/19 PR #54640 Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 PR #55925 Mono ReadWrite_Success LargeDataSentAndReceived --
7/19 PR #55925 CoreCLR ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/19 PR #55942 Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 PR #55915 Mono ReadWrite_Success MultipleReadsAndWrites --
7/19 PR #55862 Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 PR #55283 Mono ReadWrite_Success MultipleReadsAndWrites 3rd: QuicStreamTests_MockProvider.ReadWrite_Random_Success
7/19 PR #55871 Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 Official run Mono ReadWrite_Success ReadWrite_Random_Success --
7/20 PR #55564 Mono ReadWrite_Success ReadWrite_Random_Success --
7/20 PR #56003 Mono ReadWrite_Success MultipleReadsAndWrites 3rd: QuicStreamTests_MockProvider.ReadWrite_Random_Success
4th: MockQuicStreamConformanceTests.CopyToAsync_AllDataCopied
7/20 PR #55871 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/20 PR #56030 Mono ReadWrite_Success ReadWrite_Random_Success --
7/20 PR #56039 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/22 PR #56128 Mono ReadWrite_Success LargeDataSentAndReceived --
7/22 PR #56130 Mono ReadWrite_Success ReadWrite_Random_Success --
7/22 PR #56128 Mono ReadWrite_Success LargeDataSentAndReceived --
7/22 PR #56130 Mono ReadWrite_Success ReadWrite_Random_Success --
7/23 PR #56235 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/23 PR #56235 Mono ReadWrite_Success ReadWrite_Random_Success --
7/26 PR #56318 Mono ReadWrite_Success LargeDataSentAndReceived --
7/27 PR #56330 CoreCLR ReadWrite_Success LargeDataSentAndReceived --
7/28 PR #56433 Mono ReadWrite_Success ReadWrite_Random_Success --

Example from 7/28:

===========================================================================================================
/root/helix/work/workitem /root/helix/work/workitem
  Discovering: System.Net.Quic.Functional.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Quic.Functional.Tests (found 104 of 121 test cases)
  Starting:    System.Net.Quic.Functional.Tests (parallel test collections = on, max threads = 2)
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:02:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:02:16
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:04:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:04:16
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:06:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:06:16
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:08:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:08:16
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:10:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:10:16
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:12:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:12:16
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:14:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:14:16

...
[EXECUTION TIMED OUT]
Exit Code:-3Executor timed out after 900 seconds and was killed
['System.Net.Quic.Functional.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-System.Net.Quictest-run-coreTest failures in .NET Core test runs

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions