C++ exceptions thrown in SIDE_MODULES not caught (by other modules?)

[johnwason]

I am using the Pyodide project with Emscripten to compile a C++ Python library for use in the browser. I have found that exceptions are working sometimes, but other times they are not caught correctly and filter back to Python instead. If the throw and catch are in the same function, it seems to work correctly. If they are in different functions, it is not caught. Because of the use of Boost and CMake the module has to be built separately, so it is possible. I haven't found the magic combination of compiler flags to make it work. Pyodide is using side modules with dynamic loading. I looked through the generated wasm text code and the exception checks were being generated.

The original issue with more details can be found here:

pyodide/pyodide#545

Test case:

https://gist.github.com/johnwason/422b0bd9787088db3d12b422a6a835ff

[kripken]

-s EXCEPTION_DEBUG may help debug this, it will print some logging on each throw and catch etc. Specifically, I'd look at a full stack trace (in a build with --profiling) when it throws an exception you want to be caught (that option may already print the stack trace, if not, I'd just add console.log(new Error().stack) there). Then you can look up the stack to see if the expected catch is there, and if there's anything in the middle.

If you get stuck, I can take a look at such a build that's instrumented to show the stack trace etc.

[johnwason]

I have -s EXCEPTION_DEBUG=1 set but it didn't seem to do anything. See my build.sh file in the gist like above.

[kripken]

I see it on SIDE_C_FLAGS, is that just for the side module? Side modules don't have any JS for that to effect, the option would need to be on the main program.

(Btw, exceptions would need to be enabled on both the side module and the main program, if one wants to catch an exception from the other, but I think that's not the issue here from the description.)

[johnwason]

The main compilation is done by the Pyodide build scripts which download and build everything at once. This module is compiling against the environment Pyodide sets up.

The exceptions are all caught within the module so I don't think them leaking into the main module is the issue. Do I need to compile the main module with exceptions as well?

[kripken]

Oh, yes, that's the case. Exceptions support in wasm depends on JS currently, until wasm gets native exception support. So the main module needs exceptions enabled.

I'm surprised there isn't a clear error on that, I'll look into adding one.

[johnwason]

That makes sense. I will go through the (rather convoluted) build system in Pyodide and see if I can build with exceptions enabled. Thanks!

[kripken]

Cool, let me know what you find!

Btw, it looks like we should already print a clear error, "Exception catching is disabled, this exception cannot be caught" if the main module is built without exceptions. So I'm surprised you didn't see that. Perhaps it is already built with exceptions?

[johnwason]

Would that message be shown during compilation or runtime?

[johnwason]

Ok, I rebuilt the main module with the following output:

em++ -s EXPORT_NAME="'pyodide'" -o build/pyodide.asm.html src/main.bc src/jsimport.bc src/jsproxy.bc src/js2python.bc src/pyimport.bc src/pyproxy.bc src/python2js.bc src/python2js_buffer.bc src/runpython.bc src/hiwire.bc \
  -O3 -s MODULARIZE=1 cpython/installs/python-3.7.0/lib/libpython3.7.a lz4/lz4-1.8.3/lib/liblz4.a -s "BINARYEN_METHOD='native-wasm'" -s TOTAL_MEMORY=1073741824 -s ALLOW_MEMORY_GROWTH=1 -s MAIN_MODULE=1 -s EMULATED_FUNCTION_POINTERS=1 -s EMULATE_FUNCTION_POINTER_CASTS=1 -s LINKABLE=1 -s EXPORT_ALL=1 -s EXPORTED_FUNCTIONS='["___cxa_guard_acquire", "__ZNSt3__28ios_base4initEPv"]' -s WASM=1 -s SWAPPABLE_ASM_MODULE=1 -s USE_FREETYPE=1 -s USE_LIBPNG=1 -std=c++14 -Lcpython/build/sqlite-autoconf-3270200/.libs -lsqlite3 cpython/build/bzip2-1.0.2/libbz2.a -lstdc++ --memory-init-file 0 -s "BINARYEN_TRAP_MODE='clamp'" -s TEXTDECODER=0 -s LZ4=1 -s DISABLE_EXCEPTION_CATCHING=0 -s EXCEPTION_DEBUG=1 -s FORCE_FILESYSTEM=1
emcc: warning: cannot represent a NaN literal '0x555a9590bca0' with custom bit pattern in NaN-canonicalizing JS engines (e.g. Firefox and Safari) without erasing bits!
  in   %spec.select = select i1 %2, double 0x7FF8000000000000, double 0xFFF8000000000000 in _Py_dg_stdnan() 
rm build/pyodide.asm.html

This is the web console output I see when I run the module:

Python initialization complete
pyodide.asm.js:8:57200
Loading pyodide_exception_test_module from default channel
pyodide_dev.js:129:9
Loading numpy from default channel
pyodide_dev.js:129:9
Running test 0!
pyodide.asm.js:8:57200
Throw Exception!
pyodide.asm.js:8:57200
Compiled code throwing an exception, 11149552,2533368,16392
pyodide.asm.js:8:51362

Here is the wast file: https://gist.github.com/johnwason/bb0a530f41671a25e8d7760e0e9e63e3

[kripken]

The error should be shown at runtime. Strange you don't see it. What emscripten version was it built with?

Is that pasted output what you'd expect? Or is it still showing a problem? If it is, then I'd recommend as earlier to get a stack trace and investigate that.

[johnwason]

Emscripten version for Piodide is 1.38.30. There appears to be work to update the version but currently it is fixed to that version.

The out put should say: "Running test 0", "Throw Exception!", "Caught std::exception!", and then pass the exception back to Python. Currently I am not seeing "Caught Exception" or receiving an error in Python. If I throw an exception directly within the try block, it works. If I throw an exception in a function it is not caught by the enclosing catch block.

I will try enabling profiling and reply with the output.

[johnwason]

I wasn't able to get a stack trace or see anything useful in profiling. I am only seeing the small main module in the debugger in Firefox and none of the side modules. The exception seems to fall out of web assembly as soon as it is thrown.

[kripken]

Adding stack trace logging in __cxa_throw may be the simplest thing, as that is what it calls to throw.