`RangeError: Maximum call stack size exceeded` when building libopenmpt test suite with `-s WASM=2` or `-s WASM=0`

[manxorist]

Version of emscripten/emsdk:

emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.22 (a9981ae2a7dc3c45f833d0b2202f739d87ac05c8)
clang version 16.0.0 (https://github.com/llvm/llvm-project 8491d01cc385d08b8b4f5dd097239ea0009ddc63)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /home/manx/opt/emsdk/upstream/bin

Failing command line in full:

When building libopenmpt test suite (https://github.com/OpenMPT/openmpt/) with make CONFIG=emscripten TEST=1 ONLY_TEST=1 EMSCRIPTEN_TARGET=all (this implies emscripten specific options -Oz -flto -s WASM=2 -s LEGACY_VM_SUPPORT=1 -Wno-transpile -s DISABLE_EXCEPTION_CATCHING=0 -s ERROR_ON_UNDEFINED_SYMBOLS=1 -s ERROR_ON_MISSING_LIBRARIES=1 -s EXPORT_NAME="'libopenmpt'"), we are seeing:

/home/manx/opt/emsdk/upstream/emscripten/third_party/terser/terser.js:6281
          if (self instanceof AST_Scope) {
                   ^

RangeError: Maximum call stack size exceeded
    at Function.[Symbol.hasInstance] (<anonymous>)
    at AST_LabeledStatement.<anonymous> (/home/manx/opt/emsdk/upstream/emscripten/third_party/terser/terser.js:6281:20)
    at /home/manx/opt/emsdk/upstream/emscripten/third_party/terser/terser.js:6575:24
    at Array.forEach (<anonymous>)
    at display_body (/home/manx/opt/emsdk/upstream/emscripten/third_party/terser/terser.js:6566:16)
    at /home/manx/opt/emsdk/upstream/emscripten/third_party/terser/terser.js:6630:19
    at /home/manx/opt/emsdk/upstream/emscripten/third_party/terser/terser.js:6033:21
    at OutputStream.with_indent (/home/manx/opt/emsdk/upstream/emscripten/third_party/terser/terser.js:5998:40)
    at Object.with_block (/home/manx/opt/emsdk/upstream/emscripten/third_party/terser/terser.js:6032:11)
    at print_braced (/home/manx/opt/emsdk/upstream/emscripten/third_party/terser/terser.js:6629:22)
Traceback (most recent call last):
  File "/home/manx/opt/emsdk/upstream/emscripten/em++.py", line 14, in <module>
    sys.exit(emcc.run(sys.argv))
  File "/home/manx/opt/emsdk/upstream/emscripten/emcc.py", line 1226, in run
    phase_post_link(options, state, wasm_target, wasm_target, target)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/manx/opt/emsdk/upstream/emscripten/emcc.py", line 2923, in phase_post_link
    phase_binaryen(target, options, wasm_target)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/manx/opt/emsdk/upstream/emscripten/emcc.py", line 3550, in phase_binaryen
    wasm2js = building.wasm2js(wasm2js_template,
  File "/home/manx/opt/emsdk/upstream/emscripten/tools/building.py", line 1189, in wasm2js
    temp = js_optimizer(temp, passes)
  File "/home/manx/opt/emsdk/upstream/emscripten/tools/building.py", line 622, in js_optimizer
    return js_optimizer.run(filename, passes)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/manx/opt/emsdk/upstream/emscripten/tools/js_optimizer.py", line 397, in run
    return run_on_js(filename, passes, extra_info=extra_info, just_split=just_split, just_concat=just_concat)
  File "/home/manx/opt/emsdk/upstream/emscripten/tools/js_optimizer.py", line 300, in run_on_js
    filenames = shared.run_multiple_processes(commands, route_stdout_to_temp_files_suffix='js_opt.jo.js')
  File "/home/manx/opt/emsdk/upstream/emscripten/tools/shared.py", line 219, in run_multiple_processes
    raise Exception('Subprocess %d/%d failed (%s)! (cmdline: %s)' % (idx + 1, len(commands), returncode_to_str(finished_process.returncode), shlex_join(commands[idx])))
Exception: Subprocess 4/11 failed (returned 1)! (cmdline: /home/manx/opt/emsdk/node/14.18.2_64bit/bin/node /home/manx/opt/emsdk/upstream/emscripten/tools/acorn-optimizer.js /tmp/tmp_xb51efl.jsfunc_3.js minifyLocals minifyWhitespace last)
make: *** [Makefile:1310: bin/libopenmpt_test.js] Error 1
manx@appendix:~/projects/openmpt/trunk-1$ internal/fs/utils.js:332
    throw err;
    ^

Error: ENOENT: no such file or directory, open '/tmp/tmptwqcet5c.jsfunc_4.js'
    at Object.openSync (fs.js:497:3)
    at Object.readFileSync (fs.js:393:35)
    at read (/home/manx/opt/emsdk/upstream/emscripten/tools/acorn-optimizer.js:20:13)
    at Object.<anonymous> (/home/manx/opt/emsdk/upstream/emscripten/tools/acorn-optimizer.js:1824:15)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)
    at internal/main/run_main_module.js:17:47 {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: '/tmp/tmptwqcet5c.jsfunc_4.js'
}
internal/fs/utils.js:332
    throw err;
    ^

Error: ENOENT: no such file or directory, open '/tmp/tmpa2sko8el.jsfunc_5.js'
    at Object.openSync (fs.js:497:3)
    at Object.readFileSync (fs.js:393:35)
    at read (/home/manx/opt/emsdk/upstream/emscripten/tools/acorn-optimizer.js:20:13)
    at Object.<anonymous> (/home/manx/opt/emsdk/upstream/emscripten/tools/acorn-optimizer.js:1824:15)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)
    at internal/main/run_main_module.js:17:47 {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: '/tmp/tmpa2sko8el.jsfunc_5.js'
}
internal/fs/utils.js:332
    throw err;
    ^

Error: ENOENT: no such file or directory, open '/tmp/tmpsg1x_27j.jsfunc_6.js'
    at Object.openSync (fs.js:497:3)
    at Object.readFileSync (fs.js:393:35)
    at read (/home/manx/opt/emsdk/upstream/emscripten/tools/acorn-optimizer.js:20:13)
    at Object.<anonymous> (/home/manx/opt/emsdk/upstream/emscripten/tools/acorn-optimizer.js:1824:15)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)
    at internal/main/run_main_module.js:17:47 {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: '/tmp/tmpsg1x_27j.jsfunc_6.js'
}

Emscripten 3.1.21 worked fine, so this is a regression.

Building libopenmpt itself and not the test suite works also fine ('make CONFIG=emscripten TEST=0 EMSCRIPTEN_TARGET=all`).

Building the test suite with -s WASM=1 instead of -s WASM=2 -s LEGACY_VM_SUPPORT=1 -Wno-transpile works also fine (make CONFIG=emscripten TEST=1 ONLY_TEST=1 EMSCRIPTEN_TARGET=wasm).

Even if it should not be easy to fix on the emscripten side, I would appreciate hints about how I could try to work-around the issue on the libopenmpt side.

[sbc100]

That does seem odd that it regressed since terser has not been updated in a while. It would be interested in to bitsect to see exactly what caused the regression. If you have time the instructions for bisecting are here: https://emscripten.org/docs/contributing/developers_guide.html#bisecting

[manxorist]

Running bisect right now.

[manxorist]

6a58eafa5dd62f26c6bc38d8bf0a55dee97b79ff is the first bad commit
commit 6a58eafa5dd62f26c6bc38d8bf0a55dee97b79ff
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date:   Tue Sep 13 23:04:38 2022 +0000

    Roll emscripten from ea154b19446b to fff3775abca4 (3 revisions)

    https://chromium.googlesource.com/external/github.com/emscripten-core/emscripten.git/+log/ea154b19446b..fff3775abca4

    2022-09-13 sbc@chromium.org Don't pass `-fno-inline-functions` when `-Oz`/`-Os` are used (#17843)
    2022-09-13 7121787+tlively@users.noreply.github.com [Proxying][NFC] Avoid unnecessary copies of std::function (#17834)
    2022-09-13 7121787+tlively@users.noreply.github.com [WasmFS][NFC] Do not eagerly execute queue in ProxyWorker (#17835)

    If this roll has caused a breakage, revert this CL and stop the roller
    using the controls here:
    https://autoroll.skia.org/r/emscripten-emscripten-releases
    Please CC wasm-waterfall@grotations.appspotmail.com on the revert to ensure that a human
    is aware of the problem.

    To report a problem with the AutoRoller itself, please file a bug:
    https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug

    Documentation for the AutoRoller is here:
    https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

    Tbr: wasm-waterfall@grotations.appspotmail.com
    Change-Id: I217fac3bc67b0d69ff19fe04daae10f95d9d5f1e
    Reviewed-on: https://chromium-review.googlesource.com/c/emscripten-releases/+/3894706
    Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
    Bot-Commit: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>

 DEPS | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

As we are compiling with -Oz, I tried adding back -fno-inline-functions manually, and that avoids the issue, so #17843 made the problem surface, but I guess (I am no expert on emscripten internals) the real cause might probably be somewhere else.

I do not have a setup to run runtime performance numbers, but I can provide a size comparison for libopenmpt (not test suite) (with emscripten 3.1.22):

-Oz -s WASM=2:

-rw------- 1 manx manx  113716 Sep 20 17:21 libopenmpt.js
-rw-r--r-- 1 manx manx   92758 Sep 20 17:20 libopenmpt.js.mem
-rwxr-xr-x 1 manx manx 1043584 Sep 20 17:21 libopenmpt.wasm
-rw-r--r-- 1 manx manx 2009093 Sep 20 17:22 libopenmpt.wasm.js

-Oz -s WASM=2 -fno-inline-functions:

-rw------- 1 manx manx  114904 Sep 20 17:14 libopenmpt.js
-rw-r--r-- 1 manx manx   93334 Sep 20 17:13 libopenmpt.js.mem
-rwxr-xr-x 1 manx manx 1028454 Sep 20 17:14 libopenmpt.wasm
-rw-r--r-- 1 manx manx 1995643 Sep 20 17:15 libopenmpt.wasm.js

I would call at least the size difference not really significant.

We build everything with -flto.
Our unit test suite (which triggers the error) consists of multiple rather huge C++ functions in very few huge translation units that each call a lot of mostly inlineable code. I do not expect that reducing that to a small reproducible test case for this issue is feasible here.
The library (as shown in the size comparison) itself is more well-behaved medium-template-heavy C++ code.

[manxorist]

-s WASM=0 is also affected.

[sbc100]

It sounds like -fno-inline-functions is the key. For now I would just add that your project, but in the long term we probably want to run js_optimizer with a larger default stack size.

It looks like there is an FAQ entry for how to deal with running out of stack: https://emscripten.org/docs/getting_started/FAQ.html#why-do-i-get-a-stack-size-error-when-optimizing-rangeerror-maximum-call-stack-size-exceeded-or-similar. Can you confirm that adding that flag fixes the issue?

[manxorist]

I have already added -fno-inline-functions in libopenmpt.

I will try setting the stack size as a work-around tomorrow. Is there a way to set that from the em++ command line or is editing the config file the only option?

[sbc100]

You might be able to set EM_NODE_JS="/path/to/node --stack-size=xxx" in the environment.. although I'm not sure that works with arguments/spaces.

[manxorist]

Guess I still found some time today.

NODE_JS = ['node', '--stack_size=8192'] also works,

For libopenmpt, I will stick to the -fno-inline-functions work-around for now.

[manxorist]

It sounds like -fno-inline-functions is the key. For now I would just add that your project, but in the long term we probably want to run js_optimizer with a larger default stack size.

Any update on this one?

It is frankly quite annoying that, over time, I continuously have to add more and more non-standard compiler flags as work-arounds for emscripten because the default configuration of emscripten appears to less and less be able to even compile a moderately-sized C++ project successfully.

IMHO, this is kind-of a bad sign for emscripten.

[sbc100]

Sorry you were negatively effected by the -fno-inline-functions change. We do try to minimize user breakage over time,
and we certainly do want to support even the largest of C++ codebases. I'm somewhat surprised that none of our wasm2js test cases ran into this issue.

I'm also curious why nobody else has reported this. That could be because most folks don't use wasm2js, or it could be because there is something about your input program that is unique. I would love to be able to replicate this failure in test when landing the fix but I'm also OK with just landing since it seems like great prophylactic with little downside.