Compiler failing at optimization step with a big .ts file

[sebpiq]

Bug description

I am trying to compile a big file (~130k lines) that was machine generated, and is doing audio synthesis.
After a while the compiler returns the following error :

▌ Whoops, the AssemblyScript compiler has crashed during optimize :-(
▌ 
▌ Here is the stack trace hinting at the problem, perhaps it's useful?
▌ 
▌ RuntimeError: Aborted(). Build with -sASSERTIONS for more info.
▌     at x (file:///home/spiq/code/webpd/WebPd/node_modules/binaryen/index.js:6:3024)
▌     at p (file:///home/spiq/code/webpd/WebPd/node_modules/binaryen/index.js:9:3988)
▌     at wasm://wasm/016d0f36:wasm-function[92]:0x10aba
▌     at QB (file:///home/spiq/code/webpd/WebPd/node_modules/binaryen/index.js:9:108051)
▌     at wasm://wasm/016d0f36:wasm-function[1802]:0x1af2da
▌     at wasm://wasm/016d0f36:wasm-function[4982]:0x360e60
▌     at wasm://wasm/016d0f36:wasm-function[854]:0xee235
▌     at wasm://wasm/016d0f36:wasm-function[4984]:0x360ed2
▌     at DB (file:///home/spiq/code/webpd/WebPd/node_modules/binaryen/index.js:9:108308)
▌     at wasm://wasm/016d0f36:wasm-function[1388]:0x1396c6
▌ 
▌ If you see where the error is, feel free to send us a pull request. If not,
▌ please let us know: https://github.com/AssemblyScript/assemblyscript/issues
▌ 
▌ Thank you!

Some of the things I have tried :

• doing only compile without optimization step : it works
• reducing the file to a minimal test case : I couldn't. When I remove some of the code, it invariably starts working at some point.
• dividing the code in two parts and trying to compile them separately : each of the two parts compiles + optimizes without any problem, which makes me think that it is probably due to the size of the file itself.

Steps to reproduce

Run npx asc -O tmp/bla.ts with the enclosed file (zipped because github doesn't support uploading .ts files) :
bla.zip

AssemblyScript version

v0.27.22

[CountBleck]

Here is the complete error it seems:

Cannot enlarge memory, requested 2147504128 bytes, but the limit is 2147483648 bytes!
Cannot enlarge memory, requested 2147508224 bytes, but the limit is 2147483648 bytes!
Aborted(native code called abort())

▌ Whoops, the AssemblyScript compiler has crashed during optimize :-(
▌
▌ Here is the stack trace hinting at the problem, perhaps it's useful?
▌
▌ RuntimeError: Aborted(native code called abort())
▌     at abort (file:///Users/blumiere/Projects/assemblyscript/node_modules/binaryen/index.js:422:15)
▌     at _abort (file:///Users/blumiere/Projects/assemblyscript/node_modules/binaryen/index.js:3478:7)
▌     at binaryen_wasm.wasm.MixedArena::allocSpace(unsigned long, unsigned long) (wasm://wasm/binaryen_wasm.wasm-021889a6:wasm-function[813]:0x71d58)
▌     at invoke_iiii (file:///Users/blumiere/Projects/assemblyscript/node_modules/binaryen/index.js:5722:40)
▌     at binaryen_wasm.wasm.wasm::optimizeBlock(wasm::Block*, wasm::Module*, wasm::PassOptions&, wasm::BranchUtils::BranchSeekerCache&) (wasm://wasm/binaryen_wasm.wasm-021889a6:wasm-function[5986]:0x309bda)
▌     at binaryen_wasm.wasm.wasm::Walker<wasm::MergeBlocks, wasm::UnifiedExpressionVisitor<wasm::MergeBlocks, void>>::doVisitBlock(wasm::MergeBlocks*, wasm::Expression**) (wasm://wasm/binaryen_wasm.wasm-021889a6:wasm-function[5895]:0x30659f)
▌     at binaryen_wasm.wasm.wasm::Walker<wasm::MergeBlocks, wasm::UnifiedExpressionVisitor<wasm::MergeBlocks, void>>::walk(wasm::Expression*&) (wasm://wasm/binaryen_wasm.wasm-021889a6:wasm-function[5891]:0x305335)
▌     at binaryen_wasm.wasm.wasm::WalkerPass<wasm::PostWalker<wasm::MergeBlocks, wasm::UnifiedExpressionVisitor<wasm::MergeBlocks, void>>>::runOnFunction(wasm::Module*, wasm::Function*) (wasm://wasm/binaryen_wasm.wasm-021889a6:wasm-function[5893]:0x306521)
▌     at invoke_viii (file:///Users/blumiere/Projects/assemblyscript/node_modules/binaryen/index.js:5733:33)
▌     at binaryen_wasm.wasm.wasm::PassRunner::runPassOnFunction(wasm::Pass*, wasm::Function*) (wasm://wasm/binaryen_wasm.wasm-021889a6:wasm-function[2178]:0x1723f2)
▌
▌ If you see where the error is, feel free to send us a pull request. If not,
▌ please let us know: https://github.com/AssemblyScript/assemblyscript/issues
▌
▌ Thank you!

[CountBleck]

It seems to work if binaryen.js is built with a maximum of 4 GiB of memory (which is the hard maximum according to wasm-ld, likely because it's Wasm32).

Here's the gzipped bla.wasm by the way.

The time needed on my 15" mid-2014 MacBook Pro: node ../assemblyscript/bin/asc -O bla.ts -o bla.wasm 484.24s user 1.51s system 102% cpu 7:55.49 total

[sebpiq]

Thanks so much @CountBleck ! I have a couple of questions then :

• when you say "built with a maximum of 4 GiB of memory" you mean the amount of memory that's available to binaryen for optimization ?
• how would you go about detecting that error and reporting it in a more user friendly manner ?

[CountBleck]

Yes, I mean the amount of memory available to Binaryen.

To detect that error, perhaps one could check the size of HEAP8 and compare it to the maximum.

Also, it might be good to bump the maximum memory in binaryen.js to 4 GiB regardless.

[sebpiq]

Also, it might be good to bump the maximum memory in binaryen.js to 4 GiB regardless.

How do you do that ? I see only a few options are forwarded to binaryen from asc

To detect that error, perhaps one could check the size of HEAP8 and compare it to the maximum.

Same ... any pointers on how I can access these ?

[MaxGraey]

Binaryen should use add_link_flag("-sMAXIMUM_MEMORY=4gb") for that. Just need to add this flag here:
https://github.com/WebAssembly/binaryen/blob/main/CMakeLists.txt#L337

[CountBleck]

Yeah, that's basically the flag I used.

How do you do that ? I see only a few options are forwarded to binaryen from asc

That's done when binaryen.js is built, using the flag Max said.

any pointers on how I can access these ?

That would likely be done within asc and not in your code.

[MaxGraey]

Also, we could try to update binaryen.js to latest. Perhaps there are some fixes, improvements and less memory leakage or consumption.

[CountBleck]

Also, we could try to update binaryen.js to latest. Perhaps there are some fixes, improvements and less memory leakage or consumption.

I have zero experience with updating AS's binaryen.js version, which is why I haven't just stuck the flag in the action yet.

Also, I tested AS with the latest version of binaryen.js, and it doesn't work with the default 2 GiB.