[wasm] Library tests failing during linking for AOT - `SIGKILL`

[radekdoulik]

Build Browser wasm Linux Release LibraryTests_AOT fails randomly on CI

This might be caused by OOM as the wasm AOT builds are memory intensive.

CI main run fail - build, and log.

PR fail https://github.com/dotnet/runtime/runs/10048986689

  Linking with emcc with -Oz -Wl,-O0 -Wl,-lto-O0. This may take a while ...
   "/datadisks/disk1/work/A19508CB/p/build/emsdk/upstream/bin/wasm-ld" @/datadisks/disk1/work/A19508CB/t/emscripten_idqz6a8k.rsp.utf-8
   "/datadisks/disk1/work/A19508CB/p/build/emsdk/upstream/bin/wasm-emscripten-finalize" -g --dyncalls-i64 --dwarf /datadisks/disk1/work/A19508CB/w/B4A90A00/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/A19508CB/w/B4A90A00/e/wasm_build/obj/wasm/for-build/dotnet.wasm --detect-features
   "/datadisks/disk1/work/A19508CB/p/build/emsdk/node/14.18.2_64bit/bin/node" /datadisks/disk1/work/A19508CB/p/build/emsdk/upstream/emscripten/src/compiler.js /datadisks/disk1/work/A19508CB/t/tmplw9d5ysg.json
   "/datadisks/disk1/work/A19508CB/p/build/emsdk/upstream/bin/wasm-opt" --post-emscripten -Oz --low-memory-unused --zero-filled-memory --strip-producers /datadisks/disk1/work/A19508CB/w/B4A90A00/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/A19508CB/w/B4A90A00/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals
emcc : error : '/datadisks/disk1/work/A19508CB/p/build/emsdk/upstream/bin/wasm-opt --post-emscripten -Oz --low-memory-unused --zero-filled-memory --strip-producers /datadisks/disk1/work/A19508CB/w/B4A90A00/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/A19508CB/w/B4A90A00/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals' failed (received SIGKILL (-9)) [/datadisks/disk1/work/A19508CB/w/B4A90A00/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/A19508CB/p/build/wasm/WasmApp.Native.targets(431,5): error MSB3073: The command "emcc "@/datadisks/disk1/work/A19508CB/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-default.rsp" "@/datadisks/disk1/work/A19508CB/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-link.rsp" "@/datadisks/disk1/work/A19508CB/w/B4A90A00/e/wasm_build/obj/wasm/for-build/emcc-link.rsp"" exited with code 1. [/datadisks/disk1/work/A19508CB/w/B4A90A00/e/publish/ProxyProjectForAOTOnHelix.proj]

Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "dotnet.wasm -g --mvp-features --enable-mutable-globals' failed (received SIGKILL (-9))",
  "BuildRetry": false
}

Report

Build	Definition	Test	Pull Request
122829	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#80103
112529	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79671
113478	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79708
110492	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79589
115598	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79790
115574	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution
115569	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution
115554	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79775
115562	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79776
115512	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79787
115497	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution
115484	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution
115218	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79720
115452	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79790
115372	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79685
115315	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79788
115310	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79787
115246	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution
115237	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79775
115164	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79779
115160	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79778
115136	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79659
114879	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79120
114878	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution
114868	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79720
115097	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79775
115070	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79329
115057	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#78861
115050	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79169
114983	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79544
114974	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79659
114929	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution
114790	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79763
114763	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79659
114659	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79759
114347	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79613
114629	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79544
114569	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79046
114521	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79754
114507	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79685
114428	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79448
109567	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79215
114378	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79639
114323	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79744
114261	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution
114247	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution
114173	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79737
114122	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79720
114106	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79730
114071	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#78742
112612	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79677
113764	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution
113770	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79721
113740	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79719
113728	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79718
113700	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79716
113696	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution
113704	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79717
113263	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#77399
113497	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79686
113458	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79613
113268	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79659
113381	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79685
113324	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79145
113298	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79651
113283	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79698
113181	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution
112336	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79651
113158	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution
113155	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79339
112809	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79686
113093	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#78862
111861	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#77399
112813	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#76500
112793	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#78862
112670	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79544
112589	dotnet/runtime	normal-System.Runtime.Serialization.Xml.Tests.WorkItemExecution
112637	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79678
112600	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79675
112557	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79672
112315	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79659
112564	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#78860
112560	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79669
112551	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution
112553	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79672
112378	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79324
112295	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#76500
112303	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79652
112291	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79657
112269	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79472
112286	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79448
112209	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79644
112152	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#78557
112066	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79645
111843	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79334
112020	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79644
112013	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79640
112009	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79639
111952	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79046
111961	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#79339
Displaying 100 of 139 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	1	139

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Browser wasm Linux Release LibraryTests_AOT fails randomly on CI

This might be caused by OOM as the wasm AOT builds are memory intensive.

CI main run fail https://github.com/dotnet/runtime/runs/9978728829

PR fail https://github.com/dotnet/runtime/runs/10048986689

Author:	radekdoulik
Assignees:	-
Labels:	area-Build-mono, area-Infrastructure-mono
Milestone:	8.0.0

[radekdoulik]

cc @radical

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Browser wasm Linux Release LibraryTests_AOT fails randomly on CI

This might be caused by OOM as the wasm AOT builds are memory intensive.

CI main run fail - build, and log.

PR fail https://github.com/dotnet/runtime/runs/10048986689

  Linking with emcc with -Oz -Wl,-O0 -Wl,-lto-O0. This may take a while ...
   "/datadisks/disk1/work/A19508CB/p/build/emsdk/upstream/bin/wasm-ld" @/datadisks/disk1/work/A19508CB/t/emscripten_idqz6a8k.rsp.utf-8
   "/datadisks/disk1/work/A19508CB/p/build/emsdk/upstream/bin/wasm-emscripten-finalize" -g --dyncalls-i64 --dwarf /datadisks/disk1/work/A19508CB/w/B4A90A00/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/A19508CB/w/B4A90A00/e/wasm_build/obj/wasm/for-build/dotnet.wasm --detect-features
   "/datadisks/disk1/work/A19508CB/p/build/emsdk/node/14.18.2_64bit/bin/node" /datadisks/disk1/work/A19508CB/p/build/emsdk/upstream/emscripten/src/compiler.js /datadisks/disk1/work/A19508CB/t/tmplw9d5ysg.json
   "/datadisks/disk1/work/A19508CB/p/build/emsdk/upstream/bin/wasm-opt" --post-emscripten -Oz --low-memory-unused --zero-filled-memory --strip-producers /datadisks/disk1/work/A19508CB/w/B4A90A00/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/A19508CB/w/B4A90A00/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals
emcc : error : '/datadisks/disk1/work/A19508CB/p/build/emsdk/upstream/bin/wasm-opt --post-emscripten -Oz --low-memory-unused --zero-filled-memory --strip-producers /datadisks/disk1/work/A19508CB/w/B4A90A00/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/A19508CB/w/B4A90A00/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals' failed (received SIGKILL (-9)) [/datadisks/disk1/work/A19508CB/w/B4A90A00/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/A19508CB/p/build/wasm/WasmApp.Native.targets(431,5): error MSB3073: The command "emcc "@/datadisks/disk1/work/A19508CB/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-default.rsp" "@/datadisks/disk1/work/A19508CB/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-link.rsp" "@/datadisks/disk1/work/A19508CB/w/B4A90A00/e/wasm_build/obj/wasm/for-build/emcc-link.rsp"" exited with code 1. [/datadisks/disk1/work/A19508CB/w/B4A90A00/e/publish/ProxyProjectForAOTOnHelix.proj]

Author:	radekdoulik
Assignees:	-
Labels:	arch-wasm, area-Infrastructure-mono
Milestone:	8.0.0

[radical]

A lot of instances of this are being seen, on windows, and linux. And not limited to System.Runtime.Tests.

For example, Microsoft.Extensions.Configuration.UserSecrets.Tests rolling build. The last few lines of dmesg in this case showed:

[ 8659.096271] [ 2544] 1000 2544 2900 64 65536 0 0 helix.sh [ 8659.096273] [ 3303] 1000 3303 43890 6871 253952 0 0 python3 [ 8659.096274] [ 3307] 106 3307 7151 47 110592 0 0 uuidd [ 8659.096276] [ 3321] 1000 3321 64012 7801 274432 0 0 python3 [ 8659.096278] [ 3322] 1000 3322 160233 48419 688128 0 0 python3 [ 8659.096280] [ 3878] 102 3878 67335 427 172032 0 0 rsyslogd [ 8659.096281] [ 4198] 0 4198 298860 4760 253952 0 0 auoms [ 8659.096283] [ 4264] 102 4264 231174 20890 843776 0 0 mdsd [ 8659.096285] [ 4321] 0 4321 199518 5713 208896 0 0 auomscollect [ 8659.096287] [ 14954] 1000 14954 1159 17 53248 0 0 sh [ 8659.096289] [ 14955] 1000 14955 1159 16 53248 0 0 execute.sh [ 8659.096290] [ 14976] 1000 14976 2900 74 61440 0 0 bash [ 8659.096292] [ 14979] 1000 14979 68537028 34389 839680 0 0 dotnet [ 8659.096294] [ 27784] 0 27784 202011 4835 241664 0 0 azsecd [ 8659.096297] [ 27806] 0 27806 220476 4607 253952 0 0 azsecmond [ 8659.096298] [ 30266] 1000 30266 1159 17 57344 0 0 sh [ 8659.096300] [ 30268] 1000 30268 36749 25242 376832 0 0 python3 [ 8659.096302] [ 30393] 1000 30393 1795240 1757604 14221312 0 0 wasm-opt [ 8659.096303] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/helix.service,task=wasm-opt,pid=30393,uid=1000 [ 8659.096325] Out of memory: Killed process 30393 (wasm-opt) total-vm:7180960kB, anon-rss:7030416kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:13888kB oom_score_adj:0

..

oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/helix.service,task=wasm-opt,pid=30393,uid=1000
Out of memory: Killed process 30393 (wasm-opt) total-vm:7180960kB, anon-rss:7030416kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:13888kB oom_score_adj:0

wasm-opt is getting oomkill'ed in this case.

[radical]

But the failures are not limited to helix, where the VMs have 4GB memory (IIUC). One rolling building failure was hit for System.Text.RegularExpressions.Tests which AOTs on the build machine itself (with 16GB memory, IIUC).

2022-12-13T09:54:53.6640936Z   Linking with emcc with -O2. This may take a while ...
2022-12-13T09:54:57.9228444Z    "/usr/local/emscripten/emsdk/upstream/bin/wasm-ld" @/tmp/emscripten_fn3rzkk9.rsp.utf-8
2022-12-13T09:55:00.7020627Z    "/usr/local/emscripten/emsdk/upstream/bin/wasm-emscripten-finalize" -g --dyncalls-i64 --dwarf /__w/1/s/artifacts/obj/System.Text.RegularExpressions.Tests/Release/net8.0/browser-wasm/wasm/for-publish/dotnet.wasm -o /__w/1/s/artifacts/obj/System.Text.RegularExpressions.Tests/Release/net8.0/browser-wasm/wasm/for-publish/dotnet.wasm --detect-features
2022-12-13T09:57:08.1611835Z emcc : error : '/usr/local/emscripten/emsdk/upstream/bin/wasm-emscripten-finalize -g --dyncalls-i64 --dwarf /__w/1/s/artifacts/obj/System.Text.RegularExpressions.Tests/Release/net8.0/browser-wasm/wasm/for-publish/dotnet.wasm -o /__w/1/s/artifacts/obj/System.Text.RegularExpressions.Tests/Release/net8.0/browser-wasm/wasm/for-publish/dotnet.wasm --detect-features' failed (received SIGKILL (-9)) [/__w/1/s/src/libraries/System.Text.RegularExpressions/tests/FunctionalTests/System.Text.RegularExpressions.Tests.csproj::TargetFramework=net8.0]
2022-12-13T09:57:08.1952702Z ##[error]emcc(0,0): error : (NETCORE_ENGINEERING_TELEMETRY=Build) '/usr/local/emscripten/emsdk/upstream/bin/wasm-emscripten-finalize -g --dyncalls-i64 --dwarf /__w/1/s/artifacts/obj/System.Text.RegularExpressions.Tests/Release/net8.0/browser-wasm/wasm/for-publish/dotnet.wasm -o /__w/1/s/artifacts/obj/System.Text.RegularExpressions.Tests/Release/net8.0/browser-wasm/wasm/for-publish/dotnet.wasm --detect-features' failed (received SIGKILL (-9))
2022-12-13T09:57:08.2049360Z /__w/1/s/src/mono/wasm/build/WasmApp.Native.targets(431,5): error MSB3073: The command "emcc "@/__w/1/s/artifacts/bin/microsoft.netcore.app.runtime.browser-wasm/Release/runtimes/browser-wasm/native/src/emcc-default.rsp" "@/__w/1/s/artifacts/bin/microsoft.netcore.app.runtime.browser-wasm/Release/runtimes/browser-wasm/native/src/emcc-link.rsp" "@/__w/1/s/artifacts/obj/System.Text.RegularExpressions.Tests/Release/net8.0/browser-wasm/wasm/for-publish/emcc-link.rsp"" exited with code 1. 

cc @vargaz @lewing

Did something regress recently to cause these?