Why is my host app's crash stack trace overwritten/lost when Wasmtime is embedded?

[liupeidong0620]

Description

I'm embedding wasmtime as a library in my host application to run WebAssembly plugins. However, when my host program encounters a fatal signal (e.g., SIGSEGV) and should generate a core dump, the signal is being intercepted by wasmtime's internal signal handling mechanism. This prevents the default core dump generation and makes it difficult to capture the actual crash stack trace for debugging.

How can I properly configure signal handling so that different signals (especially fatal ones like SIGSEGV, SIGABRT) are handled correctly—allowing the host application to generate core dumps and print stack information as expected, while still allowing wasmtime to handle WebAssembly-related traps?

wasmtime version

wasmtime-v30.0.2-x86_64-linux-c-api.tar.xz

linux system

# uname -a
Linux 384a4bbb-d3bb-4a96-ae4e-5529fd44a702 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)

Code initialization process

int main() {
....
// init wasmtime

// Capture exception signals and print stack information
#ifndef _WIN32
  ::google::InstallFailureSignalHandler();
  ::google::InstallFailureWriter([](const char* data, int size) {
    LOG(ERROR) << "Failure:" << std::string(data, size);
    google::FlushLogFiles(google::INFO);
    google::FlushLogFiles(google::WARNING);
    google::FlushLogFiles(google::ERROR);
    google::FlushLogFiles(google::FATAL);
  });
#endif
}

The host app's coredump stack information is incorrect.

E0728 10:17:07.475342 108776 main.cpp:124] Failure:*** Aborted at 1753669027 (unix time) try "date -d @1753669027" if you are using GNU date ***
E0728 10:17:07.477213 108776 main.cpp:124] Failure:PC: @                0x0 (unknown)
E0728 10:17:07.477321 108776 main.cpp:124] Failure:*** SIGSEGV (@0xffffffffffffffef) received by PID 108417 (TID 0x7fa9931ff700) from PID 18446744073709551599; stack trace: ***
E0728 10:17:07.479700 108776 main.cpp:124] Failure:    @           0xf311f0 wasmtime::runtime::vm::sys::unix::signals::trap_handler::hfe532cb8288e83d6

[pchickey]

On Unix, Wasmtime is using SIGSEGV and SIGILL internally as part of its implementation, unless you disable it in the Config with https://docs.wasmtime.dev/api/wasmtime/struct.Config.html#method.signals_based_traps. Wasmtime will install its own signal handler over the top of a previously existing one, and if wasmtime's signal handler is unable to handle the trap (because it comes from code which is not native cranelift code), the existing handler will then be called: https://github.com/bytecodealliance/wasmtime/blob/main/crates/wasmtime/src/runtime/vm/sys/unix/signals.rs#L229. So, when you see the wasmtime signals::trap_handler on the stack in your backtrace, thats the reason why. Unfortunately, the Unix signal handler API doesn't have any mechanisms to gracefully defer to another handler without first putting wasmtime's on the stack, and we do not provide a way for wasmtime to call the previous handler via a tail call where you won't first see signals::trap_handler on the stack. It may be possible to implement such a thing for particular operating systems and architectures, but it would likely be pretty fragile, so I'd be pretty cautious about doing so.

[alexcrichton]

Is the core dump library you're using open source, and if so could you link to an example of it? The code Pat linked to should attempt to forward to the next handler, but that's either (a) not working or (b) the wasmtime signal handler is itself segfaulting and so that forwarding behavior was never hit. It may also be the case that the best option to forward to another handler is to always execute this line which is to re-run the faulting instruction with a different handler entirely.

In any case if you have an example of the core dump library you're using (or better yet a repro we could run) that'd help digging in to see what's going on. Failing that if you're able to attach a debugger (maybe by manually forcing a segfault?) it might be useful to get a backtrace at the fault, then continue to the next fault (if any) and see where execution goes and get backtraces along the way.

[liupeidong0620]

example

#  Code changes:
1. Tendis uses the wasmtime library
2. Manually generating a core dump (redis-cli -h 127.0.0.1 -p 51002 hget testkey subkey)

https://github.com/liupeidong0620/Tendis/commit/3326339ef486173e743988cf8f76db442fb9b348

# system

# cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)
# Upgrading the gcc version using scl is very simple
# gcc --version
gcc (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

# build

git clone https://github.com/liupeidong0620/Tendis.git
cd Tendis
git checkout -b dev origin/dev

git submodule update --init --recursive
mkdir build
cd build & cmake ..
make -j12

# run
cd ..
mkdir ./home/db

 ./build/bin/tendisplus ./tendisplus.conf


# redis request

./bin/redis-cli -h 127.0.0.1 -p 51002 hget testkey subkey

# generate coredump

# tail -f home/log/tendisplus.INFO
W0805 17:04:52.126920 99353 index_manager.cpp:342] index manager running...
I0805 17:04:52.127179 99354 server_entry.cpp:1971] serverCron thread starts, hz:10
I0805 17:04:52.127231 99355 server_entry.cpp:1911] bgcompact thread running...
I0805 17:04:52.127594 99320 network.cpp:380] add netIoThread, i:7, threadId:139981001582336
I0805 17:04:52.132004 99089 server_entry.cpp:870] ServerEntry::startup sucess.
E0805 17:05:49.123395 99265 main.cpp:124] Failure:*** Aborted at 1754384749 (unix time) try "date -d @1754384749" if you are using GNU date ***
E0805 17:05:49.124835 99265 main.cpp:124] Failure:PC: @                0x0 (unknown)
E0805 17:05:49.124886 99265 main.cpp:124] Failure:*** SIGSEGV (@0xffffffffffffffe8) received by PID 99089 (TID 0x7f4ffffe8700) from PID 18446744073709551592; stack trace: ***
E0805 17:05:49.125391 99265 main.cpp:124] Failure:    @          0x1c7e3f7 google::(anonymous namespace)::FailureSignalHandler()
E0805 17:05:49.126637 99265 main.cpp:124] Failure:    @           0xa2e360 wasmtime::runtime::vm::sys::unix::signals::trap_handler::hfe532cb8288e83d6

# glog lib code (main.cpp)

#ifndef _WIN32
  ::google::InstallFailureSignalHandler();
  ::google::InstallFailureWriter([](const char* data, int size) {
    LOG(ERROR) << "Failure:" << std::string(data, size);
    google::FlushLogFiles(google::INFO);
    google::FlushLogFiles(google::WARNING);
    google::FlushLogFiles(google::ERROR);
    google::FlushLogFiles(google::FATAL);
  });
#endif

@alexcrichton

[alexcrichton]

Thanks! I was able to reproduce using just the glog library. Looks like this is an issue where libunwind, the default unwinder that's using, can't unwind Rust frames from Wasmtime on the stack correctly. This is in turn due to the lack of unwinding information due to our C API artifacts being built with -Cpanic=abort. I've confirmed locally that building the C API with -Cforce-unwind-tables fixes the issue as it ensures there's unwind information for libunwind to use.

I plan on updating our default C API artifacts to include this as I think it's reasonable to have, but in the meantime you can build your own copy of the C API with RUSTFLAGS=-Cforce-unwind-tables too.