System.Runtime.Tests Outerloop tests failing with SIGKILL #66396
Description
Hit in #66387.
/root/helix/work/correlation/dotnet exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing
popd
===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
Discovered: System.Runtime.Tests (found 28 of 6309 test cases)
Starting: System.Runtime.Tests (parallel test collections = on, max threads = 2)
./RunTests.sh: line 168: 25 Killed "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Wed 09 Mar 2022 01:53:24 PM UTC ----- exit code 137 ----------------------------------------------------------
exit code 137 means SIGKILL Killed eg by kill
ulimit -c value: unlimited
[ 317.229311] 0 pages hwpoisoned
[ 317.229311] Tasks state (memory values in pages):
[ 317.229312] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 317.229316] [ 451] 0 451 23710 198 184320 0 0 systemd-journal
[ 317.229318] [ 465] 0 465 24428 44 94208 0 0 lvmetad
[ 317.229320] [ 474] 0 474 11006 348 118784 0 -1000 systemd-udevd
[ 317.229321] [ 540] 0 540 3067 267 65536 0 0 hv_kvp_daemon
[ 317.229323] [ 854] 62583 854 35446 108 180224 0 0 systemd-timesyn
[ 317.229325] [ 945] 100 945 20009 163 167936 0 0 systemd-network
[ 317.229326] [ 969] 101 969 17654 150 184320 0 0 systemd-resolve
[ 317.229328] [ 1173] 0 1173 42708 1957 233472 0 0 networkd-dispat
[ 317.229329] [ 1176] 0 1176 15502 143 167936 0 0 systemd-logind
[ 317.229331] [ 1177] 0 1177 20059 3266 196608 0 0 python3
[ 317.229333] [ 1178] 0 1178 40270 33 81920 0 0 lxcfs
[ 317.229334] [ 1183] 0 1183 7964 78 98304 0 0 cron
[ 317.229336] [ 1186] 0 1186 7084 52 98304 0 0 atd
[ 317.229337] [ 1187] 0 1187 72000 215 200704 0 0 accounts-daemon
[ 317.229339] [ 1188] 103 1188 12539 179 147456 0 -900 dbus-daemon
[ 317.229340] [ 1200] 0 1200 27605 55 122880 0 0 irqbalance
[ 317.229342] [ 1202] 0 1202 337818 5223 294912 0 -999 containerd
[ 317.229343] [ 1221] 0 1221 4104 37 73728 0 0 agetty
[ 317.229345] [ 1230] 0 1230 3723 34 69632 0 0 agetty
[ 317.229346] [ 1232] 0 1232 72221 197 204800 0 0 polkitd
[ 317.229348] [ 1370] 0 1370 1128 17 53248 0 0 none
[ 317.229349] [ 1443] 0 1443 115283 4961 270336 0 0 python3
[ 317.229351] [ 1732] 0 1732 18076 183 180224 0 -1000 sshd
[ 317.229353] [ 1899] 0 1899 410936 21841 589824 0 -500 dockerd
[ 317.229354] [ 2242] 1000 2242 2899 64 61440 0 0 helix.sh
[ 317.229356] [ 2794] 0 2794 37139 2964 192512 0 0 python3
[ 317.229357] [ 2799] 0 2799 900949 7022 303104 0 0 amacoreagent
[ 317.229358] [ 2886] 0 2886 229350 299 151552 0 0 agentlauncher
[ 317.229360] [ 2932] 0 2932 79371 69 118784 0 0 mdsdmgr
[ 317.229362] [ 3296] 1000 3296 44344 6857 253952 0 0 python3
[ 317.229363] [ 3364] 106 3364 7150 45 102400 0 0 uuidd
[ 317.229365] [ 3398] 1000 3398 63562 7108 270336 0 0 python3
[ 317.229366] [ 3399] 1000 3399 105350 10700 323584 0 0 python3
[ 317.229368] [ 3473] 102 3473 67333 139 180224 0 0 rsyslogd
[ 317.229369] [ 3605] 102 3605 150857 2983 446464 0 0 mdsd
[ 317.229371] [ 3893] 0 3893 297958 3060 249856 0 0 auoms
[ 317.229372] [ 3921] 0 3921 215737 1698 188416 0 0 auomscollect
[ 317.229374] [ 4237] 0 4237 201897 4319 241664 0 0 azsecd
[ 317.229375] [ 4248] 0 4248 177083 3317 188416 0 0 azsecmond
[ 317.229377] [ 5040] 0 5040 178212 1696 114688 0 -998 containerd-shim
[ 317.229378] [ 5062] 1000 5062 596 17 40960 0 0 helix_docker_wo
[ 317.229380] [ 5133] 1000 5133 596 17 40960 0 0 execute.sh
[ 317.229381] [ 5135] 1000 5135 1714 86 49152 0 0 bash
[ 317.229383] [ 5145] 1000 5145 2904006 1869990 15400960 0 0 dotnet
[ 317.229384] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=d6d83bd2640930892e9431ca04fa850dffadde62971abe155f3701da4bb9a1cc,mems_allowed=0,global_oom,task_memcg=/docker/d6d83bd2640930892e9431ca04fa850dffadde62971abe155f3701da4bb9a1cc,task=dotnet,pid=5145,uid=1000
[ 317.229424] Out of memory: Killed process 5145 (dotnet) total-vm:11616024kB, anon-rss:7479960kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:15040kB oom_score_adj:0
[ 317.359200] oom_reaper: reaped process 5145 (dotnet), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Waiting a few seconds for any dump to be written..
cat /proc/sys/kernel/core_pattern: /home/helixbot/dotnetbuild/dumps/core.%u.%p
cat /proc/sys/kernel/core_uses_pid: 0
cat: /proc/sys/kernel/coredump_filter: No such file or directory
cat /proc/sys/kernel/coredump_filter:
Looking around for any Linux dump..
... found no dump in /root/helix/work/workitem/e