[wasm][aot] `System.Text.Json` tests fail while linking due to OOM

[buyaa-n]

System.Text.Json.Tests failing in Build Browser wasm Release AllSubsets_Mono_AOT for [release/6.0] branch on this PR

The error log looks similar to this issue:

Linking with emcc. This may take a while ...
   "/datadisks/disk1/work/BD520A30/p/build/emsdk/upstream/bin/wasm-ld" @/datadisks/disk1/work/BD520A30/t/emscripten_l8no0ble.rsp
   "/datadisks/disk1/work/BD520A30/p/build/emsdk/upstream/bin/wasm-emscripten-finalize" --minimize-wasm-changes -g --dyncalls-i64 --dwarf /datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm --detect-features
   "/datadisks/disk1/work/BD520A30/p/build/emsdk/node/14.15.5_64bit/bin/node" /datadisks/disk1/work/BD520A30/p/build/emsdk/upstream/emscripten/src/compiler.js /datadisks/disk1/work/BD520A30/t/tmpa4attyye.txt
   "/datadisks/disk1/work/BD520A30/p/build/emsdk/upstream/bin/wasm-opt" --post-emscripten --no-exit-runtime -Oz --low-memory-unused --zero-filled-memory --strip-producers /datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals
emcc : error : '/datadisks/disk1/work/BD520A30/p/build/emsdk/upstream/bin/wasm-opt --post-emscripten --no-exit-runtime -Oz --low-memory-unused --zero-filled-memory --strip-producers /datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals' failed (received SIGKILL (-9)) [/datadisks/disk1/work/BD520A30/w/99F6090D/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/BD520A30/p/build/wasm/WasmApp.Native.targets(374,5): error MSB3073: The command "emcc "@/datadisks/disk1/work/BD520A30/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-default.rsp" "@/datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/emcc-link.rsp"" exited with code 1. [/datadisks/disk1/work/BD520A30/w/99F6090D/e/publish/ProxyProjectForAOTOnHelix.proj]

Same failure in #61491

Runfo Tracking Issue: System.Text.Json.Tests failing in Build Browser wasm Release AllSubsets_Mono_AOT leg

Build	Definition	Kind	Run Name	Console	Core Dump	Test Results	Run Client
1681152	runtime	PR 67126	net7.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1681152	runtime	PR 67126	net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1656324	runtime	PR 65614	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1655763	runtime	PR 66438	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1654725	runtime	Rolling	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1654719	runtime	PR 66408	net7.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1654719	runtime	PR 66408	net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1654324	runtime	PR 66373	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1654259	runtime	Rolling	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1653373	runtime	PR 65898	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1653029	runtime	PR 66373	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1652926	runtime	PR 66144	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1652810	runtime	Rolling	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1652772	runtime	PR 65537	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1652119	runtime	Rolling	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1651977	runtime	PR 66353	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1650022	runtime	PR 66315	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1649325	runtime	PR 66301	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1649187	runtime	PR 66299	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1646435	runtime	PR 66230	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1646135	runtime	PR 66225	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1644313	runtime	Rolling	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1643540	runtime	Rolling	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1643051	runtime	PR 66144	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1641227	runtime	PR 66021	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1641169	runtime	PR 66106	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1638414	runtime	PR 66021	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1636812	runtime	PR 65979	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1632648	runtime	PR 65898	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1630724	runtime	PR 65857	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1630090	runtime	PR 65768	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py
1629211	runtime	Rolling	net6.0-Browser-Release-wasm-Mono_Release-normal-Ubuntu.1804.Amd64.Open	console.log			runclient.py

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
1	1	30

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

---

System.Text.Json.Tests failing in Build Browser wasm Release AllSubsets_Mono_AOT for [release/6.0] branch on this PR

The error log looks similar to this issue:

Linking with emcc. This may take a while ...
   "/datadisks/disk1/work/BD520A30/p/build/emsdk/upstream/bin/wasm-ld" @/datadisks/disk1/work/BD520A30/t/emscripten_l8no0ble.rsp
   "/datadisks/disk1/work/BD520A30/p/build/emsdk/upstream/bin/wasm-emscripten-finalize" --minimize-wasm-changes -g --dyncalls-i64 --dwarf /datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm --detect-features
   "/datadisks/disk1/work/BD520A30/p/build/emsdk/node/14.15.5_64bit/bin/node" /datadisks/disk1/work/BD520A30/p/build/emsdk/upstream/emscripten/src/compiler.js /datadisks/disk1/work/BD520A30/t/tmpa4attyye.txt
   "/datadisks/disk1/work/BD520A30/p/build/emsdk/upstream/bin/wasm-opt" --post-emscripten --no-exit-runtime -Oz --low-memory-unused --zero-filled-memory --strip-producers /datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals
emcc : error : '/datadisks/disk1/work/BD520A30/p/build/emsdk/upstream/bin/wasm-opt --post-emscripten --no-exit-runtime -Oz --low-memory-unused --zero-filled-memory --strip-producers /datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals' failed (received SIGKILL (-9)) [/datadisks/disk1/work/BD520A30/w/99F6090D/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/BD520A30/p/build/wasm/WasmApp.Native.targets(374,5): error MSB3073: The command "emcc "@/datadisks/disk1/work/BD520A30/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-default.rsp" "@/datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/emcc-link.rsp"" exited with code 1. [/datadisks/disk1/work/BD520A30/w/99F6090D/e/publish/ProxyProjectForAOTOnHelix.proj]

Author:	buyaa-n
Assignees:	-
Labels:	untriaged, area-Infrastructure-mono
Milestone:	-

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

---

System.Text.Json.Tests failing in Build Browser wasm Release AllSubsets_Mono_AOT for [release/6.0] branch on this PR

The error log looks similar to this issue:

Linking with emcc. This may take a while ...
   "/datadisks/disk1/work/BD520A30/p/build/emsdk/upstream/bin/wasm-ld" @/datadisks/disk1/work/BD520A30/t/emscripten_l8no0ble.rsp
   "/datadisks/disk1/work/BD520A30/p/build/emsdk/upstream/bin/wasm-emscripten-finalize" --minimize-wasm-changes -g --dyncalls-i64 --dwarf /datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm --detect-features
   "/datadisks/disk1/work/BD520A30/p/build/emsdk/node/14.15.5_64bit/bin/node" /datadisks/disk1/work/BD520A30/p/build/emsdk/upstream/emscripten/src/compiler.js /datadisks/disk1/work/BD520A30/t/tmpa4attyye.txt
   "/datadisks/disk1/work/BD520A30/p/build/emsdk/upstream/bin/wasm-opt" --post-emscripten --no-exit-runtime -Oz --low-memory-unused --zero-filled-memory --strip-producers /datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals
emcc : error : '/datadisks/disk1/work/BD520A30/p/build/emsdk/upstream/bin/wasm-opt --post-emscripten --no-exit-runtime -Oz --low-memory-unused --zero-filled-memory --strip-producers /datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals' failed (received SIGKILL (-9)) [/datadisks/disk1/work/BD520A30/w/99F6090D/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/BD520A30/p/build/wasm/WasmApp.Native.targets(374,5): error MSB3073: The command "emcc "@/datadisks/disk1/work/BD520A30/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-default.rsp" "@/datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/emcc-link.rsp"" exited with code 1. [/datadisks/disk1/work/BD520A30/w/99F6090D/e/publish/ProxyProjectForAOTOnHelix.proj]

Same failure in #61491

Author:	buyaa-n
Assignees:	-
Labels:	arch-wasm, untriaged, area-Infrastructure-mono
Milestone:	-

[lewing]

/datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals' failed (received SIGKILL (-9)) [/datadisks/disk1/work/BD520A30/w/99F6090D/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/BD520A30/p/build/wasm/WasmApp.Native.targets(374,5): error MSB3073: The command "emcc "@/datadisks/disk1/work/BD520A30/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-default.rsp" "@/datadisks/disk1/work/BD520A30/w/99F6090D/e/wasm_build/obj/wasm/for-build/emcc-link.rsp"" exited with code 1. 

This is the VM killing the test because it ran out of memory.

cc @steveisok

[lewing]

cc @ilyas1974

[ericstj]

Still happening in release/6.0. https://dev.azure.com/dnceng/public/_build/results?buildId=1541389&view=ms.vss-test-web.build-test-results-tab&runId=43596408&resultId=158013&paneView=debug

  Linking with emcc. This may take a while ...
   "/datadisks/disk1/work/AF620989/p/build/emsdk/upstream/bin/wasm-ld" @/datadisks/disk1/work/AF620989/t/emscripten_u6kd2dfl.rsp
   "/datadisks/disk1/work/AF620989/p/build/emsdk/upstream/bin/wasm-emscripten-finalize" --minimize-wasm-changes -g --dyncalls-i64 --dwarf /datadisks/disk1/work/AF620989/w/A26E0953/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/AF620989/w/A26E0953/e/wasm_build/obj/wasm/for-build/dotnet.wasm --detect-features
   "/datadisks/disk1/work/AF620989/p/build/emsdk/node/14.15.5_64bit/bin/node" /datadisks/disk1/work/AF620989/p/build/emsdk/upstream/emscripten/src/compiler.js /datadisks/disk1/work/AF620989/t/tmpjx6y3suh.txt
   "/datadisks/disk1/work/AF620989/p/build/emsdk/upstream/bin/wasm-opt" --post-emscripten --no-exit-runtime -Oz --low-memory-unused --zero-filled-memory --strip-producers /datadisks/disk1/work/AF620989/w/A26E0953/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/AF620989/w/A26E0953/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals
emcc : error : '/datadisks/disk1/work/AF620989/p/build/emsdk/upstream/bin/wasm-opt --post-emscripten --no-exit-runtime -Oz --low-memory-unused --zero-filled-memory --strip-producers /datadisks/disk1/work/AF620989/w/A26E0953/e/wasm_build/obj/wasm/for-build/dotnet.wasm -o /datadisks/disk1/work/AF620989/w/A26E0953/e/wasm_build/obj/wasm/for-build/dotnet.wasm -g --mvp-features --enable-mutable-globals' failed (received SIGKILL (-9)) [/datadisks/disk1/work/AF620989/w/A26E0953/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/AF620989/p/build/wasm/WasmApp.Native.targets(375,5): error MSB3073: The command "emcc "@/datadisks/disk1/work/AF620989/p/build/microsoft.netcore.app.runtime.browser-wasm/runtimes/browser-wasm/native/src/emcc-default.rsp" "@/datadisks/disk1/work/AF620989/w/A26E0953/e/wasm_build/obj/wasm/for-build/emcc-link.rsp"" exited with code 1. [/datadisks/disk1/work/AF620989/w/A26E0953/e/publish/ProxyProjectForAOTOnHelix.proj]

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-62191-merge-b78860dd399545deaa/System.Text.Json.Tests/1/console.0337f9dc.log?sv=2019-07-07&se=2022-01-27T18%3A59%3A12Z&sr=c&sp=rl&sig=wtlCKkyFcU8KzsGfG7bVdNZqf4yTj7OvdPz3dy5M0e8%3D

[danmoseley]

In my case #63507 (comment) it was a 1 hour timeout. The one above would seem to be as well, although the step only took 45 mins not an hour.

Is it possible to make wasm-opt more verbose (when in CI) so we might get an idea how far it gets ( --debug piped to a file that gets uploaded? ) Or, reduce the optimization level from -Oz so it's faster? (Of course I know nothing of this stuff, just wondering how we can make it actionable..)

[danmoseley]

Again #63527

[steveisok]

@danmoseley I may be butchering the explanation somewhat, but STJ is making LLVM work a bit too hard (generated code size is pretty large) and sometimes we run out of memory. The only two options I know of are 1.) run outside of a docker container and see if the problem goes away or 2.) bump the memory. I had pushed for 2 outright, but we're going to have to make the case by trying 1.

@lewing Correct me if I am wrong, but changing optimization levels I don't think will help all that much.