Linker crash in Writer::run()

[sergeyext]

Initially I faced this problem with
libc.a(scalbnf.o): attempt to add bitcode file after LTO and added -Wl,-u,scalbnf to the linker command line as advised in that issue discussion. After that I got the linker crash. Probably the issue should go to llvm upstream because the crash handler says 'PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/'.

Version of emscripten/emsdk:

emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.42 (6ede0b8fc1c979bb206148804bfb48b472ccc3da)
clang version 17.0.0 (https://github.com/llvm/llvm-project f3b64887de61020c09404bfee97b2fadd30df10a)
Target: wasm32-unknown-emscripten
Thread model: posix

Failing command line in full:

cd /path/to/project/client/cmake/cmake-build-web-new/PlayClient && /usr/local/bin/cmake -E cmake_link_script CMakeFiles/engine.dir/link.txt --verbose=1
/home/sergey/code/emsdk-new/upstream/emscripten/em++ -fwasm-exceptions -flto -fno-rtti -s USE_OGG=1 -s USE_VORBIS=1 -s USE_ZLIB=1 -Wno-invalid-utf8 -O2 -g -DNDEBUG -DuseAssertLevel1  -fwasm-exceptions -flto -s DEMANGLE_SUPPORT=1 -s USE_OGG=1 -s USE_VORBIS=1 -s USE_ZLIB=1 -s ABORTING_MALLOC=0 --js-library /path/to/project/client/cmake/../jsLibs/../../render/Drawer/Platform/Gl/WebglOesDrawBuffersIndexed.js -Wl,-u,scalbnf -s TOTAL_MEMORY=480MB -s NO_EXIT_RUNTIME=1 -s EXPORTED_FUNCTIONS="['_main', '_platformWork', '_mouseLeave', '_mouseReturn']" -s NO_FILESYSTEM=1 -s USE_GLFW=3 -s WASM=1 -s EXPORTED_RUNTIME_METHODS='["ccall", "cwrap"]' -s USE_FREETYPE=1 -s USE_ZLIB=1 -s USE_OGG=1 -s USE_VORBIS=1 -s MAX_WEBGL_VERSION=2 --js-library /path/to/project/client/cmake/../jsLibs/Render3D.js --js-library /path/to/project/client/cmake/../jsLibs/Helper.js --js-library /path/to/project/client/cmake/../../render/Content/Image/Parsing/WebHelper.js --js-library /path/to/project/client/cmake/../jsLibs/InputManager.js --js-library /path/to/project/client/cmake/../jsLibs/../../render/Drawer/Platform/Gl/WebglVersionSupport.js --pre-js /path/to/project/client/cmake/../jsLibs/pre.js @CMakeFiles/engine.dir/objects1.rsp -o engine.js @CMakeFiles/engine.dir/linklibs.rsp

It's output:

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: /home/sergey/code/emsdk-new/upstream/bin/wasm-ld @/tmp/emscripten__vfbaa6u.rsp.utf-8
 #0 0x0000560d038f0228 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/sergey/code/emsdk-new/upstream/bin/wasm-ld+0xef0228)
 #1 0x0000560d038edb4e llvm::sys::RunSignalHandlers() (/home/sergey/code/emsdk-new/upstream/bin/wasm-ld+0xeedb4e)
 #2 0x0000560d038f0e35 SignalHandler(int) Signals.cpp:0:0
 #3 0x00007f32471c88e0 __restore_rt (/lib64/libc.so.6+0x3d8e0)
 #4 0x0000560d03ce7e1e lld::wasm::(anonymous namespace)::Writer::run() Writer.cpp:0:0
 #5 0x0000560d03cdbf18 lld::wasm::writeResult() (/home/sergey/code/emsdk-new/upstream/bin/wasm-ld+0x12dbf18)
 #6 0x0000560d03cb49b2 lld::wasm::(anonymous namespace)::LinkerDriver::linkerMain(llvm::ArrayRef<char const*>) Driver.cpp:0:0
 #7 0x0000560d03cb0a20 lld::wasm::link(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, bool, bool) (/home/sergey/code/emsdk-new/upstream/bin/wasm-ld+0x12b0a20)
 #8 0x0000560d038f2c9f lld::unsafeLldMain(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, llvm::ArrayRef<lld::DriverDef>, bool) (/home/sergey/code/emsdk-new/upstream/bin/wasm-ld+0xef2c9f)
 #9 0x0000560d0383c472 lld_main(int, char**, llvm::ToolContext const&) (/home/sergey/code/emsdk-new/upstream/bin/wasm-ld+0xe3c472)
#10 0x0000560d0383cb2c main (/home/sergey/code/emsdk-new/upstream/bin/wasm-ld+0xe3cb2c)
#11 0x00007f32471b25b0 __libc_start_call_main (/lib64/libc.so.6+0x275b0)
#12 0x00007f32471b2679 __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x27679)
#13 0x0000560d037b69aa _start (/home/sergey/code/emsdk-new/upstream/bin/wasm-ld+0xdb69aa)
em++: error: '/home/sergey/code/emsdk-new/upstream/bin/wasm-ld @/tmp/emscripten__vfbaa6u.rsp.utf-8' failed (received SIGSEGV (-11))
make[2]: *** [PlayClient/CMakeFiles/engine.dir/build.make:10773: PlayClient/engine.js] Error 1
make[2]: leaving directory «/path/to/project/client/cmake/cmake-build-web-new»
make[1]: *** [CMakeFiles/Makefile2:369: PlayClient/CMakeFiles/engine.dir/all] Error 2
make[1]: leaving directory «/path/to/project/client/cmake/cmake-build-web-new»
make: *** [Makefile:103: all] Error 2

Full link command and output with -v appended:

cd /path/to/project/client/cmake/cmake-build-web-new/PlayClient && /usr/local/bin/cmake -E cmake_link_script CMakeFiles/engine.dir/link.txt --verbose=1
/home/sergey/code/emsdk-new/upstream/emscripten/em++ -fwasm-exceptions -flto -fno-rtti -s USE_OGG=1 -s USE_VORBIS=1 -s USE_ZLIB=1 -Wno-invalid-utf8 -O2 -g -DNDEBUG -DuseAssertLevel1  -fwasm-exceptions -flto -s DEMANGLE_SUPPORT=1 -s USE_OGG=1 -s USE_VORBIS=1 -s USE_ZLIB=1 -s ABORTING_MALLOC=0 --js-library /path/to/project/client/cmake/../jsLibs/../../render/Drawer/Platform/Gl/WebglOesDrawBuffersIndexed.js -Wl,-u,scalbnf -s TOTAL_MEMORY=480MB -s NO_EXIT_RUNTIME=1 -s EXPORTED_FUNCTIONS="['_main', '_platformWork', '_mouseLeave', '_mouseReturn']" -s NO_FILESYSTEM=1 -s USE_GLFW=3 -s WASM=1 -s EXPORTED_RUNTIME_METHODS='["ccall", "cwrap"]' -s USE_FREETYPE=1 -s USE_ZLIB=1 -s USE_OGG=1 -s USE_VORBIS=1 -s MAX_WEBGL_VERSION=2 --js-library /path/to/project/client/cmake/../jsLibs/Render3D.js --js-library /path/to/project/client/cmake/../jsLibs/Helper.js --js-library /path/to/project/client/cmake/../../render/Content/Image/Parsing/WebHelper.js --js-library /path/to/project/client/cmake/../jsLibs/InputManager.js --js-library /path/to/project/client/cmake/../jsLibs/../../render/Drawer/Platform/Gl/WebglVersionSupport.js --pre-js /path/to/project/client/cmake/../jsLibs/pre.js @CMakeFiles/engine.dir/objects1.rsp -o engine.js @CMakeFiles/engine.dir/linklibs.rsp -v

Output:

 "/home/sergey/code/emsdk-new/upstream/bin/clang" --version
 "/home/sergey/code/emsdk-new/upstream/bin/wasm-ld" @/tmp/emscripten_00aiz_j1.rsp.utf-8

the rest lines are as above starting with 'PLEASE submit a bug report....'

Contents of /tmp/emscripten_00aiz_j1.rsp.utf-8:

.... list of projects's object file paths ...
-L/home/sergey/code/emsdk-new/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto
/home/sergey/code/emsdk-new/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libfreetype-wasm-sjlj.a
/home/sergey/code/emsdk-new/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libvorbis.a
/home/sergey/code/emsdk-new/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libogg.a
/home/sergey/code/emsdk-new/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libz.a
--whole-archive
-lGL-webgl2
--no-whole-archive
-lal
-lhtml5
-lstubs
-lnoexit
-lc
-ldlmalloc
-lcompiler_rt-wasm-sjlj
-lc++-except
-lc++abi-except
-lunwind-except
-lsockets
-mllvm
-combiner-global-alias-analysis=false
-mllvm
-wasm-enable-sjlj
-mllvm
-disable-lsr
-mllvm
-wasm-enable-eh
-mllvm
-exception-model=wasm
/tmp/tmp7uaemuillibemscripten_js_symbols.so
--export-if-defined=main
--export-if-defined=platformWork
--export-if-defined=mouseLeave
--export-if-defined=mouseReturn
--export-if-defined=__start_em_asm
--export-if-defined=__stop_em_asm
--export-if-defined=__start_em_lib_deps
--export-if-defined=__stop_em_lib_deps
--export-if-defined=__start_em_js
--export-if-defined=__stop_em_js
--export-if-defined=__main_argc_argv
--export=__cxa_demangle
--export=free
--export=stackSave
--export=stackRestore
--export=stackAlloc
--export=__errno_location
--export=__get_temp_ret
--export=__set_temp_ret
--export=__trap
--export=__wasm_call_ctors
--export-table
-z
stack-size=65536
--initial-memory=503316480
--no-entry
--max-memory=503316480
--global-base=1024

[sbc100]

If possible could you link with -Wl,--reproduce=repro.tar and attach the resulting archive.

[sergeyext]

@sbc100 I linked with -Wl,--reproduce=repro.tar and examined the archive. Unfortunately, I'm not allowed to share it.

[sbc100]

Does installing 3.1.42-asserts (i.e. ./emsdk install 3.1.42-asserts) give you any more information in the backtrace?

Does this only effect the LTO build?

[Mintyboi]

I'm hitting into this too.

@sbc100 Nope I do not see this issue in a non-LTO build.
Using the -asserts variant of the toolchain (3.1.41-asserts) gives these additional info:

wasm-ld: /b/s/w/ir/cache/builder/emscripten-releases/llvm-project/llvm/include/llvm/Support/Casting.h:578: decltype(auto) llvm::cast(From *) [To = lld::wasm::DefinedFunction, From = const lld::wasm::FunctionSymbol]: Assertion `isa<To>(Val) && "cast<Ty>() argument of incompatible type!"' failed.

[sbc100]

@Mintyboi did you also add -Wl,-ufoo arguments to work around an attempt to add bitcode file after LTO error?

Do you have a repro case you can share?

Does the problem go away if you build your application object files with -flto but don't specify -flo at link time (i.e. don't LTO the standard libraries, just your application code).

[Mintyboi]

did you also add -Wl,-ufoo arguments to work around an attempt to add bitcode file after LTO error?

Nope, I did not use the ufoo flag.
Unfortunately, I do not have a sample reproducer at the moment. I'm working on using bugpoint to see if I can get one.

Does the problem go away if you build your application object files with -flto but don't specify -flo at link time (i.e. don't LTO the standard libraries, just your application code).

Let me try that real quick and get back to you

[Mintyboi]

Does the problem go away if you build your application object files with -flto but don't specify -flo at link time (i.e. don't LTO the standard libraries, just your application code).

The error still persists

[sbc100]

Aside from a repro a good next step would be try to pin point when this regressed. There are instructions for how to bisect on emsdk build here: https://emscripten.org/docs/contributing/developers_guide.html#bisecting.

Its not completely obvious to me that this is the exact same issue as OP, but its certainly possible.

[jeremiahar]

I'm hitting into this too.

@sbc100 Nope I do not see this issue in a non-LTO build. Using the -asserts variant of the toolchain (3.1.41-asserts) gives these additional info:

wasm-ld: /b/s/w/ir/cache/builder/emscripten-releases/llvm-project/llvm/include/llvm/Support/Casting.h:578: decltype(auto) llvm::cast(From *) [To = lld::wasm::DefinedFunction, From = const lld::wasm::FunctionSymbol]: Assertion `isa<To>(Val) && "cast<Ty>() argument of incompatible type!"' failed.

I also ran into this problem. I linked with -Wl,--reproduce=repro.tar and used the tarball to run a bisect across the emscripten versions. I also am not able to share it and do not have a minimal reproduction. From my tests, 3.1.25 links while 3.1.26 does not. I ran a bisect against the emscripten-releases repository and discovered that 4e2ffe94b04dbadfbca1687ab458d306b3414d13 is the first broken commit.

I tested originally with a custom built clang 16.0.6 with -DCMAKE_BUILD_TYPE=RelWithDebInfo and -DLLVM_ENABLE_ASSERTIONS=ON. Excuse the Bazel sandbox stuff (it's also not the official emscripten bazel toolchain.)

***stack trace is from 16.0.6***

wasm-ld: /docker_nvme/bazel/sandbox/bazel-sandbox.215/linux-sandbox/1/execroot/redacted/tmp/src/llvm/include/llvm/Support/Casting.h:579: decltype(auto) llvm::cast(From*) [with To = lld::wasm::DefinedFunction; From = const lld::wasm::FunctionSymbol]: Assertion `isa<To>(Val) && "cast<Ty>() argument of incompatible type!"' failed.
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: /docker_nvme/bazel/sandbox/bazel-sandbox.215/linux-sandbox/898/execroot/redacted/bazel-out/linux_x64-opt-exec-2B5CBBC6-ST-f25b3b66d621/bin/toolchains/cc/emscripten/cache/sysroot/bin/wasm-ld @/tmp/emscripten_nf4uryoy.rsp.utf-8
 #0 0x0000563165006abe llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /docker_nvme/bazel/sandbox/bazel-sandbox.215/linux-sandbox/1/execroot/redacted/tmp/src/llvm/lib/Support/Unix/Signals.inc:571:3
 #1 0x0000563165004354 llvm::sys::RunSignalHandlers() /docker_nvme/bazel/sandbox/bazel-sandbox.215/linux-sandbox/1/execroot/redacted/tmp/src/llvm/lib/Support/Signals.cpp:104:20
 #2 0x00005631650044c6 SignalHandler(int) /docker_nvme/bazel/sandbox/bazel-sandbox.215/linux-sandbox/1/execroot/redacted/tmp/src/llvm/lib/Support/Unix/Signals.inc:412:1
 #3 0x00007fabda17cab0 (/usr/lib/libc.so.6+0x39ab0)
 #4 0x00007fabda1cc26c (/usr/lib/libc.so.6+0x8926c)
 #5 0x00007fabda17ca08 raise (/usr/lib/libc.so.6+0x39a08)
 #6 0x00007fabda165538 abort (/usr/lib/libc.so.6+0x22538)
 #7 0x00007fabda16545c (/usr/lib/libc.so.6+0x2245c)
 #8 0x00007fabda1753d6 (/usr/lib/libc.so.6+0x323d6)
 #9 0x00005631653f7ac0 lld::wasm::FunctionSymbol::setFunctionIndex(unsigned int) /docker_nvme/bazel/sandbox/bazel-sandbox.215/linux-sandbox/1/execroot/redacted/tmp/src/lld/wasm/Symbols.cpp:261:55
#10 0x000056316540e79a createCallCtorsFunction /docker_nvme/bazel/sandbox/bazel-sandbox.215/linux-sandbox/1/execroot/redacted/tmp/src/lld/wasm/Writer.cpp:1393:19
#11 0x000056316540e79a lld::wasm::(anonymous namespace)::Writer::run() /docker_nvme/bazel/sandbox/bazel-sandbox.215/linux-sandbox/1/execroot/redacted/tmp/src/lld/wasm/Writer.cpp:1628:28
#12 0x0000563165410290 lld::wasm::writeResult() /docker_nvme/bazel/sandbox/bazel-sandbox.215/linux-sandbox/1/execroot/redacted/tmp/src/lld/wasm/Writer.cpp:1723:22
#13 0x00005631653d4097 lld::wasm::(anonymous namespace)::LinkerDriver::linkerMain(llvm::ArrayRef<char const*>) (.constprop.0) /docker_nvme/bazel/sandbox/bazel-sandbox.215/linux-sandbox/1/execroot/redacted/tmp/src/lld/wasm/Driver.cpp:1239:1
#14 0x00005631653d4e3e std::_Vector_base<lld::wasm::InputFile*, std::allocator<lld::wasm::InputFile*>>::~_Vector_base() /docker_nvme/bazel/bazel/external/linux_x64_gcc_prebuilt/usr/include/c++/13.1.0/bits/stl_vector.h:367:45
#15 0x00005631653d4e3e std::vector<lld::wasm::InputFile*, std::allocator<lld::wasm::InputFile*>>::~vector() /docker_nvme/bazel/bazel/external/linux_x64_gcc_prebuilt/usr/include/c++/13.1.0/bits/stl_vector.h:733:7
#16 0x00005631653d4e3e ~LinkerDriver /docker_nvme/bazel/sandbox/bazel-sandbox.215/linux-sandbox/1/execroot/redacted/tmp/src/lld/wasm/Driver.cpp:70:7
#17 0x00005631653d4e3e lld::wasm::link(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, bool, bool) /docker_nvme/bazel/sandbox/bazel-sandbox.215/linux-sandbox/1/execroot/redacted/tmp/src/lld/wasm/Driver.cpp:100:3
#18 0x0000563164f8e822 lldMain(int, char const**, llvm::raw_ostream&, llvm::raw_ostream&, bool) /docker_nvme/bazel/sandbox/bazel-sandbox.215/linux-sandbox/1/execroot/redacted/tmp/src/lld/tools/lld/lld.cpp:168:16
#19 0x0000563164f8ec56 lld_main(int, char**) /docker_nvme/bazel/sandbox/bazel-sandbox.215/linux-sandbox/1/execroot/redacted/tmp/src/lld/tools/lld/lld.cpp:230:19
#20 0x00007fabda166850 (/usr/lib/libc.so.6+0x23850)
#21 0x00007fabda16690a __libc_start_main (/usr/lib/libc.so.6+0x2390a)
#22 0x0000563164f8c085 _start /tmp/reproducible/gcc/glibc/csu/../sysdeps/x86_64/start.S:117:0
emcc: error: '/docker_nvme/bazel/sandbox/bazel-sandbox.215/linux-sandbox/898/execroot/redacted/bazel-out/linux_x64-opt-exec-2B5CBBC6-ST-f25b3b66d621/bin/toolchains/cc/emscripten/cache/sysroot/bin/wasm-ld @/tmp/emscripten_nf4uryoy.rsp.utf-8' failed (received SIGABRT (-6))

Hopefully it's enough information to have a better idea of what went wrong.

[jeremiahar]

I bisected lld between effd75bda4b1a9b26e554c1cda3e3b4c72fa0aa8 and f81f0cb75a2808a67d2662f044ad07628fc9d900 and determined that 89ddcff1d2d6e9f4de78f3a563a8b1987bf7ea8f is the first broken llvm commit.

wasm-ld: /home/jeremiah/projects/llvm-project/llvm/include/llvm/Support/Casting.h:578: decltype(auto) llvm::cast(From*) [with To = lld::wasm::DefinedFunction; From = const lld::wasm::FunctionSymbol]: Assertion `isa<To>(Val) && "cast<Ty>() argument of incompatible type!"' failed.
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments:
...

 #0 0x00005631fd9b3d14 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /home/jeremiah/projects/llvm-project/llvm/lib/Support/Unix/Signals.inc:573:3
 #1 0x00005631fd9b1b54 llvm::sys::RunSignalHandlers() /home/jeremiah/projects/llvm-project/llvm/lib/Support/Signals.cpp:104:20
 #2 0x00005631fd9b1cc6 SignalHandler(int) /home/jeremiah/projects/llvm-project/llvm/lib/Support/Unix/Signals.inc:407:1
 #3 0x00007f5a54162ab0 (/usr/lib/libc.so.6+0x39ab0)
 #4 0x00007f5a541b226c (/usr/lib/libc.so.6+0x8926c)
 #5 0x00007f5a54162a08 raise (/usr/lib/libc.so.6+0x39a08)
 #6 0x00007f5a5414b538 abort (/usr/lib/libc.so.6+0x22538)
 #7 0x00007f5a5414b45c (/usr/lib/libc.so.6+0x2245c)
 #8 0x00007f5a5415b3d6 (/usr/lib/libc.so.6+0x323d6)
 #9 0x00005631fdd5ba87 llvm::optional_detail::OptionalStorage<unsigned int, true>::value() const & /home/jeremiah/projects/llvm-project/llvm/include/llvm/Support/Casting.h:578:3
#10 0x00005631fdd5ba87 llvm::Optional<unsigned int>::value() const & /home/jeremiah/projects/llvm-project/llvm/include/llvm/ADT/Optional.h:313:60
#11 0x00005631fdd5ba87 lld::wasm::InputFunction::getFunctionIndex() const /home/jeremiah/projects/llvm-project/lld/wasm/InputChunks.h:277:65
#12 0x00005631fdd5ba87 lld::wasm::FunctionSymbol::getFunctionIndex() const /home/jeremiah/projects/llvm-project/lld/wasm/Symbols.cpp:254:39
#13 0x00005631fdd721a6 createCallCtorsFunction /home/jeremiah/projects/llvm-project/lld/wasm/Writer.cpp:1392:19
#14 0x00005631fdd721a6 lld::wasm::(anonymous namespace)::Writer::run() /home/jeremiah/projects/llvm-project/lld/wasm/Writer.cpp:1627:28
#15 0x00005631fdd73900 llvm::SmallDenseMap<llvm::StringRef, lld::wasm::OutputSegment*, 4u, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, lld::wasm::OutputSegment*>>::deallocateBuckets() /home/jeremiah/projects/llvm-project/llvm/include/llvm/ADT/DenseMap.h:1174:5
#16 0x00005631fdd73900 llvm::SmallDenseMap<llvm::StringRef, lld::wasm::OutputSegment*, 4u, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, lld::wasm::OutputSegment*>>::~SmallDenseMap() /home/jeremiah/projects/llvm-project/llvm/include/llvm/ADT/DenseMap.h:934:22
#17 0x00005631fdd73900 ~Writer /home/jeremiah/projects/llvm-project/lld/wasm/Writer.cpp:50:7
#18 0x00005631fdd73900 lld::wasm::writeResult() /home/jeremiah/projects/llvm-project/lld/wasm/Writer.cpp:1722:22
#19 0x00005631fdd39001 lld::wasm::(anonymous namespace)::LinkerDriver::linkerMain(llvm::ArrayRef<char const*>) (.constprop.0) /home/jeremiah/projects/llvm-project/lld/wasm/Driver.cpp:1147:1
#20 0x00005631fdd39933 std::_Vector_base<lld::wasm::InputFile*, std::allocator<lld::wasm::InputFile*>>::~_Vector_base() /usr/include/c++/13.1.1/bits/stl_vector.h:367:45
#21 0x00005631fdd39933 std::vector<lld::wasm::InputFile*, std::allocator<lld::wasm::InputFile*>>::~vector() /usr/include/c++/13.1.1/bits/stl_vector.h:735:7
#22 0x00005631fdd39933 ~LinkerDriver /home/jeremiah/projects/llvm-project/lld/wasm/Driver.cpp:68:7
#23 0x00005631fdd39933 lld::wasm::link(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, bool, bool) /home/jeremiah/projects/llvm-project/lld/wasm/Driver.cpp:98:3
#24 0x00005631fd92f572 lldMain(int, char const**, llvm::raw_ostream&, llvm::raw_ostream&, bool) /home/jeremiah/projects/llvm-project/lld/tools/lld/lld.cpp:166:16
#25 0x00005631fd92f8fe lld_main(int, char**) /home/jeremiah/projects/llvm-project/lld/tools/lld/lld.cpp:228:19
#26 0x00007f5a5414c850 (/usr/lib/libc.so.6+0x23850)
#27 0x00007f5a5414c90a __libc_start_main (/usr/lib/libc.so.6+0x2390a)
#28 0x00005631fd92d005 _start (/home/jeremiah/projects/llvm-project/build/bin/wasm-ld+0x500005)

[kripken]

Looks like llvm/llvm-project@89ddcff was reverted here: llvm/llvm-project@885e610

But if we are still seeing the problem, perhaps it was re-landed? It would be good to find that commit and give the author a testcase showing the problem.

[jeremiahar]

Indeed, blame for some of the modified code points here llvm/llvm-project@12050a3 . I originally encountered the problem when refactoring some code to use inline variables and inline template specializations. Perhaps I can find a minimal test case with some luck.

[jeremiahar]

I was able to create a simplified test case using 3.1.43-asserts:

CMakeLists.txt:

cmake_minimum_required(VERSION 3.13)

set(CMAKE_EXECUTABLE_SUFFIX_CXX .html)
set(CMAKE_CXX_STANDARD 17)

add_executable(lto_bug dummy.cpp)

add_library(main
  main.cpp
)

add_library(lib1
  lib1.cpp
)

target_link_libraries(lto_bug
  PRIVATE

  main
  lib1
)

main.cpp

int DoIt()
{
    return 0; 
}
inline int Unused0 = DoIt();

int GetUnused();

int main(int, char **)
{
    return GetUnused();
}

lib1.cpp:

int DoIt();
inline int Unused0 = DoIt();

int GetUnused()
{
    return Unused0;
}

dummy.cpp:

// Dummy file for cmake's add_executable

Compilation commands:

[jeremiah@jeremiah build]$ ~/emsdk/upstream/emscripten/emcmake cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-flto" ../
configure: cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS=-flto ../ -DCMAKE_TOOLCHAIN_FILE=/home/jeremiah/emsdk/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -DCMAKE_CROSSCOMPILING_EMULATOR=/home/jeremiah/emsdk/node/16.20.0_64bit/bin/node
CMake Warning (dev) in CMakeLists.txt:
  No project() command is present.  The top-level CMakeLists.txt file must
  contain a literal, direct call to the project() command.  Add a line of
  code such as

    project(ProjectName)

  near the top of the file, but after cmake_minimum_required().

  CMake is pretending there is a "project(Project)" command on the first
  line.
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) in CMakeLists.txt:
  cmake_minimum_required() should be called prior to this top-level project()
  call.  Please see the cmake-commands(7) manual for usage documentation of
  both commands.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring done (0.5s)
-- Generating done (0.0s)
-- Build files have been written to: /home/jeremiah/projects/emscripten_lto_bug/build
[jeremiah@jeremiah build]$ make -j64 VERBOSE=1
/usr/bin/cmake -S/home/jeremiah/projects/emscripten_lto_bug -B/home/jeremiah/projects/emscripten_lto_bug/build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /home/jeremiah/projects/emscripten_lto_bug/build/CMakeFiles /home/jeremiah/projects/emscripten_lto_bug/build//CMakeFiles/progress.marks
make  -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/home/jeremiah/projects/emscripten_lto_bug/build'
make  -f CMakeFiles/lib1.dir/build.make CMakeFiles/lib1.dir/depend
make  -f CMakeFiles/main.dir/build.make CMakeFiles/main.dir/depend
make[2]: Entering directory '/home/jeremiah/projects/emscripten_lto_bug/build'
cd /home/jeremiah/projects/emscripten_lto_bug/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/jeremiah/projects/emscripten_lto_bug /home/jeremiah/projects/emscripten_lto_bug /home/jeremiah/projects/emscripten_lto_bug/build /home/jeremiah/projects/emscripten_lto_bug/build /home/jeremiah/projects/emscripten_lto_bug/build/CMakeFiles/lib1.dir/DependInfo.cmake --color=
make[2]: Entering directory '/home/jeremiah/projects/emscripten_lto_bug/build'
cd /home/jeremiah/projects/emscripten_lto_bug/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/jeremiah/projects/emscripten_lto_bug /home/jeremiah/projects/emscripten_lto_bug /home/jeremiah/projects/emscripten_lto_bug/build /home/jeremiah/projects/emscripten_lto_bug/build /home/jeremiah/projects/emscripten_lto_bug/build/CMakeFiles/main.dir/DependInfo.cmake --color=
Dependencies file "CMakeFiles/lib1.dir/lib1.cpp.o.d" is newer than depends file "/home/jeremiah/projects/emscripten_lto_bug/build/CMakeFiles/lib1.dir/compiler_depend.internal".
Consolidate compiler generated dependencies of target lib1
make[2]: Leaving directory '/home/jeremiah/projects/emscripten_lto_bug/build'
make  -f CMakeFiles/lib1.dir/build.make CMakeFiles/lib1.dir/build
Dependencies file "CMakeFiles/main.dir/main.cpp.o.d" is newer than depends file "/home/jeremiah/projects/emscripten_lto_bug/build/CMakeFiles/main.dir/compiler_depend.internal".
Consolidate compiler generated dependencies of target main
make[2]: Leaving directory '/home/jeremiah/projects/emscripten_lto_bug/build'
make  -f CMakeFiles/main.dir/build.make CMakeFiles/main.dir/build
make[2]: Entering directory '/home/jeremiah/projects/emscripten_lto_bug/build'
make[2]: Nothing to be done for 'CMakeFiles/lib1.dir/build'.
make[2]: Leaving directory '/home/jeremiah/projects/emscripten_lto_bug/build'
make[2]: Entering directory '/home/jeremiah/projects/emscripten_lto_bug/build'
make[2]: Nothing to be done for 'CMakeFiles/main.dir/build'.
make[2]: Leaving directory '/home/jeremiah/projects/emscripten_lto_bug/build'
[ 33%] Built target lib1
[ 66%] Built target main
make  -f CMakeFiles/lto_bug.dir/build.make CMakeFiles/lto_bug.dir/depend
make[2]: Entering directory '/home/jeremiah/projects/emscripten_lto_bug/build'
cd /home/jeremiah/projects/emscripten_lto_bug/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/jeremiah/projects/emscripten_lto_bug /home/jeremiah/projects/emscripten_lto_bug /home/jeremiah/projects/emscripten_lto_bug/build /home/jeremiah/projects/emscripten_lto_bug/build /home/jeremiah/projects/emscripten_lto_bug/build/CMakeFiles/lto_bug.dir/DependInfo.cmake --color=
Dependencies file "CMakeFiles/lto_bug.dir/dummy.cpp.o.d" is newer than depends file "/home/jeremiah/projects/emscripten_lto_bug/build/CMakeFiles/lto_bug.dir/compiler_depend.internal".
Consolidate compiler generated dependencies of target lto_bug
make[2]: Leaving directory '/home/jeremiah/projects/emscripten_lto_bug/build'
make  -f CMakeFiles/lto_bug.dir/build.make CMakeFiles/lto_bug.dir/build
make[2]: Entering directory '/home/jeremiah/projects/emscripten_lto_bug/build'
[ 83%] Linking CXX executable lto_bug.html
/usr/bin/cmake -E cmake_link_script CMakeFiles/lto_bug.dir/link.txt --verbose=1
/home/jeremiah/emsdk/upstream/emscripten/em++ -flto -O3 -DNDEBUG @CMakeFiles/lto_bug.dir/objects1.rsp -o lto_bug.html @CMakeFiles/lto_bug.dir/linkLibs.rsp
wasm-ld: /b/s/w/ir/cache/builder/emscripten-releases/llvm-project/llvm/include/llvm/Support/Casting.h:578: decltype(auto) llvm::cast(From *) [To = lld::wasm::DefinedFunction, From = const lld::wasm::FunctionSymbol]: Assertion `isa<To>(Val) && "cast<Ty>() argument of incompatible type!"' failed.
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: /home/jeremiah/emsdk/upstream/bin/wasm-ld -o lto_bug.wasm CMakeFiles/lto_bug.dir/dummy.cpp.o libmain.a liblib1.a -L/home/jeremiah/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto -lGL -lal -lhtml5 -lstubs -lnoexit -lc -ldlmalloc -lcompiler_rt -lc++-noexcept -lc++abi-noexcept -lsockets -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr /tmp/tmp2ow3pfaelibemscripten_js_symbols.so --strip-debug --export-if-defined=main --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-if-defined=__main_argc_argv --export=stackSave --export=stackRestore --export=stackAlloc --export=__errno_location --export=__get_temp_ret --export=__set_temp_ret --export=__wasm_call_ctors --export-table -z stack-size=65536 --initial-memory=16777216 --no-entry --max-memory=16777216 --global-base=1024
 #0 0x00007f58210121e8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/jeremiah/emsdk/upstream/bin/../lib/libLLVM-17git.so+0x1c121e8)
 #1 0x00007f582100feae llvm::sys::RunSignalHandlers() (/home/jeremiah/emsdk/upstream/bin/../lib/libLLVM-17git.so+0x1c0feae)
 #2 0x00007f58210128c5 SignalHandler(int) Signals.cpp:0:0
 #3 0x00007f581f00cab0 (/usr/lib/libc.so.6+0x39ab0)
 #4 0x00007f581f05c26c (/usr/lib/libc.so.6+0x8926c)
 #5 0x00007f581f00ca08 raise (/usr/lib/libc.so.6+0x39a08)
 #6 0x00007f581eff5538 abort (/usr/lib/libc.so.6+0x22538)
 #7 0x00007f581eff545c (/usr/lib/libc.so.6+0x2245c)
 #8 0x00007f581f0053d6 (/usr/lib/libc.so.6+0x323d6)
 #9 0x0000561afea34177 (/home/jeremiah/emsdk/upstream/bin/wasm-ld+0x4f8177)
#10 0x0000561afea3d957 lld::wasm::(anonymous namespace)::Writer::run() Writer.cpp:0:0
#11 0x0000561afea35846 lld::wasm::writeResult() (/home/jeremiah/emsdk/upstream/bin/wasm-ld+0x4f9846)
#12 0x0000561afea0e185 lld::wasm::(anonymous namespace)::LinkerDriver::linkerMain(llvm::ArrayRef<char const*>) Driver.cpp:0:0
#13 0x0000561afea0b10a lld::wasm::link(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, bool, bool) (/home/jeremiah/emsdk/upstream/bin/wasm-ld+0x4cf10a)
#14 0x0000561afe6e7552 lld::unsafeLldMain(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, llvm::ArrayRef<lld::DriverDef>, bool) (/home/jeremiah/emsdk/upstream/bin/wasm-ld+0x1ab552)
#15 0x0000561afe6e6699 lld_main(int, char**, llvm::ToolContext const&) (/home/jeremiah/emsdk/upstream/bin/wasm-ld+0x1aa699)
#16 0x0000561afe6e6b71 main (/home/jeremiah/emsdk/upstream/bin/wasm-ld+0x1aab71)
#17 0x00007f581eff6850 (/usr/lib/libc.so.6+0x23850)
#18 0x00007f581eff690a __libc_start_main (/usr/lib/libc.so.6+0x2390a)
#19 0x0000561afe6e624a _start (/home/jeremiah/emsdk/upstream/bin/wasm-ld+0x1aa24a)
em++: error: '/home/jeremiah/emsdk/upstream/bin/wasm-ld -o lto_bug.wasm CMakeFiles/lto_bug.dir/dummy.cpp.o libmain.a liblib1.a -L/home/jeremiah/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto -lGL -lal -lhtml5 -lstubs -lnoexit -lc -ldlmalloc -lcompiler_rt -lc++-noexcept -lc++abi-noexcept -lsockets -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr /tmp/tmp2ow3pfaelibemscripten_js_symbols.so --strip-debug --export-if-defined=main --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-if-defined=__main_argc_argv --export=stackSave --export=stackRestore --export=stackAlloc --export=__errno_location --export=__get_temp_ret --export=__set_temp_ret --export=__wasm_call_ctors --export-table -z stack-size=65536 --initial-memory=16777216 --no-entry --max-memory=16777216 --global-base=1024' failed (received SIGABRT (-6))
make[2]: *** [CMakeFiles/lto_bug.dir/build.make:101: lto_bug.html] Error 1
make[2]: Leaving directory '/home/jeremiah/projects/emscripten_lto_bug/build'
make[1]: *** [CMakeFiles/Makefile2:88: CMakeFiles/lto_bug.dir/all] Error 2
make[1]: Leaving directory '/home/jeremiah/projects/emscripten_lto_bug/build'
make: *** [Makefile:91: all] Error 2
[jeremiah@jeremiah build]$ 

[stanhuff]

I have a similar issue that goes away when I remove the -flto link flag. I've been using 3.1.28 for months just fine and some time during the the last two weeks some versions of my code that built just fine two weeks ago are now crashing during link. I'm aware my OS has been updating the clang tool chain, but I've not been updating emscripten.

I tried updating to a number of emscripten versions between 3.1.28 and 3.1.43 and all of them have the same result for me (SIGSEGV) except the one below (3.1.43-asserts), which aborts rather than faulting, presumably due to the assert build.

 #0 0x00007f0475e121e8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/stan/projects/emsdk/upstream/bin/../lib/libLLVM-17git.so+0x1c121e8)
 #1 0x00007f0475e0feae llvm::sys::RunSignalHandlers() (/home/stan/projects/emsdk/upstream/bin/../lib/libLLVM-17git.so+0x1c0feae)
 #2 0x00007f0475e128c5 SignalHandler(int) Signals.cpp:0:0
 #3 0x00007f0473e42520 (/lib/x86_64-linux-gnu/libc.so.6+0x42520)
 #4 0x00007f0473e96a7c __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
 #5 0x00007f0473e96a7c __pthread_kill_internal ./nptl/pthread_kill.c:78:10
 #6 0x00007f0473e96a7c pthread_kill ./nptl/pthread_kill.c:89:10
 #7 0x00007f0473e42476 gsignal ./signal/../sysdeps/posix/raise.c:27:6
 #8 0x00007f0473e287f3 abort ./stdlib/abort.c:81:7
 #9 0x00007f0473e2871b _nl_load_domain ./intl/loadmsgcat.c:1177:9
#10 0x00007f0473e39e96 (/lib/x86_64-linux-gnu/libc.so.6+0x39e96)
#11 0x000055c43cd3a177 (/home/stan/projects/emsdk/upstream/bin/wasm-ld+0x4f8177)
#12 0x000055c43cd43957 lld::wasm::(anonymous namespace)::Writer::run() Writer.cpp:0:0
#13 0x000055c43cd3b846 lld::wasm::writeResult() (/home/stan/projects/emsdk/upstream/bin/wasm-ld+0x4f9846)
#14 0x000055c43cd14185 lld::wasm::(anonymous namespace)::LinkerDriver::linkerMain(llvm::ArrayRef<char const*>) Driver.cpp:0:0
#15 0x000055c43cd1110a lld::wasm::link(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, bool, bool) (/home/stan/projects/emsdk/upstream/bin/wasm-ld+0x4cf10a)
#16 0x000055c43c9ed552 lld::unsafeLldMain(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, llvm::ArrayRef<lld::DriverDef>, bool) (/home/stan/projects/emsdk/upstream/bin/wasm-ld+0x1ab552)
#17 0x000055c43c9ec699 lld_main(int, char**, llvm::ToolContext const&) (/home/stan/projects/emsdk/upstream/bin/wasm-ld+0x1aa699)
#18 0x000055c43c9ecb71 main (/home/stan/projects/emsdk/upstream/bin/wasm-ld+0x1aab71)
#19 0x00007f0473e29d90 __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#20 0x00007f0473e29e40 call_init ./csu/../csu/libc-start.c:128:20
#21 0x00007f0473e29e40 __libc_start_main ./csu/../csu/libc-start.c:379:5
#22 0x000055c43c9ec24a _start (/home/stan/projects/emsdk/upstream/bin/wasm-ld+0x1aa24a)
emcc: error: '/home/stan/projects/emsdk/upstream/bin/wasm-ld -o build-wasm/native.wasm.wasm build-wasm/src/browserClient/main.o build-wasm/glue.o build-wasm/src/browserClient/browserClient.a build-wasm/src/browserClient/browserPlatform/browserPlatform.a build-wasm/src/browserClient/ui/ui.a build-wasm/src/browserClient/commands/commands.a build-wasm/src/world/audio/audio.a build-wasm/src/world/platforms/platforms.a build-wasm/src/world/threading/threading.a build-wasm/lib/loguru/loguru.a build-wasm/lib/bullet/libBulletCollision.a build-wasm/lib/bullet/libBulletDynamics.a build-wasm/lib/bullet/libLinearMath.a build-wasm/lib/openSimplexNoise/openSimplexNoise.a build-wasm/src/test/test.a build-wasm/src/util/util.a build-wasm/src/gen/prefabs/prefabs.a build-wasm/src/gen/gen.a build-wasm/src/external/external.a build-wasm/src/math/math.a build-wasm/src/world/renderTypes/renderTypes.a build-wasm/src/world/blobIO/blobIO.a build-wasm/src/world/items/items.a build-wasm/src/world/actions/actions.a build-wasm/src/world/chunkJobs/chunkJobs.a build-wasm/src/world/physics/physics.a build-wasm/src/world/blockTypes/blockTypes.a build-wasm/src/world/entity/entity.a build-wasm/src/world/collisionTypes/collisionTypes.a build-wasm/src/world/inputControllers/inputControllers.a build-wasm/src/world/world.a -L/home/stan/projects/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto /home/stan/projects/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/crtbegin.o -lGL-mt -lal -lhtml5 -lbulkmemory -lstubs -lnoexit -lc-mt -ldlmalloc-mt -lcompiler_rt-wasm-sjlj-mt -lc++-mt-except -lc++abi-mt-except -lunwind-mt-except -lsockets-mt -mllvm -combiner-global-alias-analysis=false -mllvm -wasm-enable-sjlj -mllvm -disable-lsr -mllvm -wasm-enable-eh -mllvm -exception-model=wasm /tmp/tmppm8z5xm0libemscripten_js_symbols.so --import-memory --shared-memory --export-if-defined=malloc --export-if-defined=setShowInteractiveListener --export-if-defined=free --export-if-defined=main --export-if-defined=_emscripten_thread_init --export-if-defined=_emscripten_thread_exit --export-if-defined=_emscripten_thread_crashed --export-if-defined=_emscripten_tls_init --export-if-defined=pthread_self --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-if-defined=__main_argc_argv --export=__cxa_demangle --export=free --export=stackSave --export=stackRestore --export=stackAlloc --export=__errno_location --export=emscripten_dispatch_to_thread_ --export=_emscripten_thread_free_data --export=emscripten_main_runtime_thread_id --export=emscripten_main_thread_process_queued_calls --export=_emscripten_run_in_main_runtime_thread_js --export=emscripten_stack_set_limits --export=__trap --export=__wasm_call_ctors --export-table --growable-table -z stack-size=65536 --initial-memory=2147418112 --no-entry --max-memory=2147418112 --global-base=1024' failed (received SIGABRT (-6))