Skip to content

assert in ProcessWaitState on Linux arm64  #74795

Closed
@wfurt

Description

@wfurt

Disabled tests:

Crash in System.Net.Requests.Tests Work Item - Last 7 days in Runfo:

  • 9/8 - 142x failures out of which 115x are on arm64-Mono
Day Run Details
9/7 Rolling run 6968 Core Dump - net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open
9/6 Rolling run 6173 2x Core Dump - net7.0-Linux-Release-arm64-Mono_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open & net7.0-Linux-Release-arm64-Mono_release-(Debian.11.Arm64.Open)Ubuntu.1804.Armarch.Open
9/4 Rolling run 4317 Code Rump - net7.0-Linux-Release-arm64-CoreCLR_release-(Ubuntu.2204.Arm64.Open)Ubuntu.1804.ArmArch.Open
8/30 Rolling run 1975580 2x Core Dump - net7.0-Linux-Release-arm64-Mono_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open & net7.0-Linux-Release-arm64-Mono_release-(Debian.11.Arm64.Open)Ubuntu.1804.Armarch.Open
8/29 Rolling run 1973318 Code Dump - net7.0-Linux-Release-arm64-Mono_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open
8/28 Rolling run 1972808 Core Dump - net7.0-Linux-Release-arm64-Mono_release-(Debian.11.Arm64.Open)Ubuntu.1804.Armarch.Open
8/28 Rolling run 1972465 Core Dump - net7.0-Linux-Release-arm64-Mono_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open
8/26 Rolling run 1970381 Core Dump - net7.0-Linux-Release-arm64-CoreCLR_release-(Ubuntu.2204.Arm64.Open)Ubuntu.1804.ArmArch.Open

Related to #69125.
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-aaacaf8e0a7f46c4ad/System.Net.Requests.Tests/1/console.1429bd54.log?%3Fhelixlogtype%3Dresult

/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Net.Requests.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Requests.Tests (found 349 of 367 test cases)
  Starting:    System.Net.Requests.Tests (parallel test collections = on, max threads = 4)
Process terminated. Error while reaping child. errno = 10
   at System.Environment.FailFast(System.String)
   at System.Diagnostics.ProcessWaitState.TryReapChild(Boolean)
   at System.Diagnostics.ProcessWaitState.CheckChildren(Boolean, Boolean)
   at System.Diagnostics.Process.OnSigChild(Int32, Int32)
./RunTests.sh: line 168:    21 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE

dump: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-aaacaf8e0a7f46c4ad/System.Net.Requests.Tests/1/core.1001.21

errno 10 -> ENOCHILD

> clrstack -a
OS Thread Id: 0x1c (0)
        Child SP               IP Call Site
0000007EBADC5460 0000007fa357f200 [HelperMethodFrame_1OBJ: 0000007ebadc5460] System.Environment.FailFast(System.String)
0000007EBADC55E0 0000007F64861C34 System.Diagnostics.ProcessWaitState.TryReapChild(Boolean) [/_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/ProcessWaitState.Unix.cs @ 589]
    PARAMETERS:
        this (0x0000007EBADC5630) = 0x000000156f599f80
        configureConsole (0x0000007EBADC562C) = 0x0000000000000001
    LOCALS:
        0x0000007EBADC5620 = 0x000000156f599fc8
        0x0000007EBADC5618 = 0x0000000000000001
        0x0000007EBADC5610 = 0x0000000060230000
        0x0000007EBADC560C = 0x00000000ffffffff
        0x0000007EBADC5608 = 0x0000000000000001
        0x0000007EBADC5600 = 0x000000000000000a

0000007EBADC5640 0000007F648610C8 System.Diagnostics.ProcessWaitState.CheckChildren(Boolean, Boolean) [/_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/ProcessWaitState.Unix.cs @ 614]
    PARAMETERS:
        reapAll (0x0000007EBADC5724) = 0x0000000000000000
        configureConsole (0x0000007EBADC5720) = 0x0000000000000001
    LOCALS:
        0x0000007EBADC5718 = 0x000000156f42a280
        0x0000007EBADC5710 = 0x0000000000000001
        0x0000007EBADC570C = 0x0000000000000000
        0x0000007EBADC5708 = 0x000000000000003c
        0x0000007EBADC5700 = 0x000000156f599f80
        0x0000007EBADC56F8 = 0x0000000000000000
        0x0000007EBADC56F0 = 0x0000000000000000
        0x0000007EBADC56E8 = 0x0000000000000000
        0x0000007EBADC56C0 = 0x0000000000000000
        0x0000007EBADC56B0 = 0x0000000000000000
        0x0000007EBADC56A8 = 0x0000000000000000
        0x0000007EBADC5690 = 0x0000000000000000
        0x0000007EBADC5688 = 0x0000000000000000
        0x0000007EBADC5680 = 0x0000000000000000

0000007EBADC5730 0000007F64860B40 System.Diagnostics.Process.OnSigChild(Int32, Int32) [/_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs @ 1104]
    PARAMETERS:
        reapAll = <no data>
        configureConsole (<CLR reg>) = 0x0000000000000001
    LOCALS:
        <CLR reg> = 0x0000000000000001
        <no data>
        <no data>

> dumpobj 0x000000156f599f80
Name:        System.Diagnostics.ProcessWaitState
MethodTable: 0000007f6480b9a8
EEClass:     0000007f64827760
Tracked Type: false
Size:        72(0x48) bytes
File:        /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/System.Diagnostics.Process.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
0000007f62d49320  4000179        8        System.Object  0 instance 000000156f599fc8 _gate
0000007f63b4e388  400017a       20         System.Int32  1 instance               60 _processId
0000007f63b4ae88  400017b       28       System.Boolean  1 instance                1 _isChild
0000007f63b4ae88  400017c       29       System.Boolean  1 instance                1 _usesTerminal
0000007f63e07350  400017d       10 ...eading.Tasks.Task  0 instance 0000000000000000 _waitInProgress
0000007f63b4e388  400017e       24         System.Int32  1 instance                2 _outstandingRefCount
0000007f63b4ae88  400017f       2a       System.Boolean  1 instance                0 _exited
0000007f63c7cc10  4000180       2c ...Private.CoreLib]]  1 instance 000000156f599fac _exitCode
0000007f63f72c08  4000181       38      System.DateTime  1 instance 000000156f599fb8 _exitTime
0000007f645345b0  4000182       18 ....ManualResetEvent  0 instance 000000156f599fe0 _exitedEvent
0000007f6480d118  4000177       60 ...nostics.Process]]  0   static 000000156f42a230 s_processWaitStates
0000007f6480d118  4000178       68 ...nostics.Process]]  0   static 000000156f42a280 s_childProcessWaitStates

and with (some) symbols

> clrstack -i -a



Dumping managed stack and managed variables using ICorDebug.
=============================================================================
Child SP         IP               Call Site
0000007EBADC4A60 0000007fa357f200 [NativeStackFrame]
0000007EBADC5460 (null) [Internal call: 0000007EBADC5460]
0000007EBADC55E0 0000007f64861c34 [DEFAULT] [hasThis] Boolean System.Diagnostics.ProcessWaitState.TryReapChild(Boolean) (/root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/System.Diagnostics.Process.dll)

PARAMETERS:
  + System.Diagnostics.ProcessWaitState this @ 0x156f599f80
  + bool configureConsole  = true

LOCALS:
  + (Error 0x80004005 retrieving local variable 'local_0')
  + (Error 0x80004005 retrieving local variable 'local_1')
  + int exitCode  = 1612906496
  + int waitResult  = -1
  + (Error 0x80004005 retrieving local variable 'local_4')
  + int errorCode  = 10

0000007EBADC5640 0000007f648610c8 [DEFAULT] Void System.Diagnostics.ProcessWaitState.CheckChildren(Boolean,Boolean) (/root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/System.Diagnostics.Process.dll)

PARAMETERS:
  + bool reapAll  = false
  + bool configureConsole  = true

LOCALS:
  + (Error 0x80004005 retrieving local variable 'local_0')
  + (Error 0x80004005 retrieving local variable 'local_1')
  + bool checkAll  = false
  + int pid  = 60
  + System.Diagnostics.ProcessWaitState pws @ 0x156f599f80
  + int errorCode  = 0
  + System.Diagnostics.ProcessWaitState firstToRemove = null
  + System.Collections.Generic.List`1&lt;System.Diagnostics.ProcessWaitState&gt; additionalToRemove = null
  + (Error 0x80004005 retrieving local variable 'local_8')
  + System.Collections.Generic.KeyValuePair`2&lt;int,System.Diagnostics.ProcessWaitState&gt; kv @ 0x7ebadc56b0
  + System.Diagnostics.ProcessWaitState pws = null
  + (Error 0x80004005 retrieving local variable 'local_11')
  + System.Diagnostics.ProcessWaitState pws = null
  + (Error 0x80004005 retrieving local variable 'local_13')

0000007EBADC5730 0000007f64860b40 [DEFAULT] I4 System.Diagnostics.Process.OnSigChild(I4,I4) (/root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/System.Diagnostics.Process.dll)

PARAMETERS:
  + (Error 0x80131304 retrieving parameter 'reapAll')
  + int configureConsole  = 1

LOCALS:
  + bool childrenUsingTerminalPre  = true
  + (Error 0x80004005 retrieving local variable 'childrenUsingTerminalPost')
  + (Error 0x80004005 retrieving local variable 'local_2')

0000007EBADC5790 0000007f60218260 [NativeStackFrame]
Stack walk complete.
=============================================================================

cc: @tmds

{ "ErrorMessage":"Error while reaping child. errno = 10" } 

Report

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 0

Metadata

Metadata

Assignees

Labels

Known Build ErrorUse this to report build issues in the .NET Helix tabarch-arm64area-System.Net.Quicblocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'os-linuxLinux OS (any supported distro)

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions