Attempts to catch_unwind (on Linux) result in SIGABRT

[stuhood]

Bug Description

Some code within a pyfunction is panicing, but rather than this resulting in a PyException via #797, it leads to a SIGABRT.

A related issue was previously reported here: pantsbuild/pants#13526 (comment) ... the fix for #1990 probably removed a panic, but other panics still seem to trigger the SIGABRT.

Steps to Reproduce

Unfortunately, the triggering panic is non-deterministic, and seemingly only occurs on Linux. Directly triggering a panic within the relevant pyfunction, properly results in a pyo3_runtime.PanicException. I'll try to report back with more information and a repro. See pantsbuild/pants#12831 (comment).

Backtrace

[Current thread is 1 (Thread 0x7fe99a924640 (LWP 3724751))]
(gdb) backtrace
#0  raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:49
#1  <signal handler called>
#2  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49
#3  0x00007fe99fb64864 in __GI_abort () at abort.c:79
#4  0x00007fe99fbc749c in __libc_message (action=do_abort, fmt=0x7fe99fcec792 "%s", fmt=0x7fe99fcec792 "%s", action=do_abort) at ../sysdeps/posix/libc_fatal.c:155
#5  0x00007fe99fbc77b0 in __GI___libc_fatal (message=message@entry=0x7fe99fe9c118 "FATAL: exception not rethrown\n") at ../sysdeps/posix/libc_fatal.c:164
#6  0x00007fe99fe96cf6 in unwind_cleanup (reason=<optimized out>, exc=<optimized out>) at unwind.c:115
#7  0x00007fe99e77730f in panic_unwind::real_imp::cleanup () at library/panic_unwind/src/gcc.rs:78
#8  panic_unwind::__rust_panic_cleanup () at library/panic_unwind/src/lib.rs:97
#9  0x00007fe99de1873d in std::panicking::try::cleanup () at library/std/src/panicking.rs:384
#10 0x00007fe99dfe449e in std::panicking::try::do_catch<pyo3::callback::handle_panic::{closure#0}, core::result::Result<*mut pyo3::ffi::object::PyObject, pyo3::err::PyErr>> (payload=0x7fe99a924cb0, 
    data=<optimized out>) at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:428
#11 std::panicking::try<core::result::Result<*mut pyo3::ffi::object::PyObject, pyo3::err::PyErr>, pyo3::callback::handle_panic::{closure#0}> (f=...)
    at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:367
#12 std::panic::catch_unwind<pyo3::callback::handle_panic::{closure#0}, core::result::Result<*mut pyo3::ffi::object::PyObject, pyo3::err::PyErr>> (f=...)
    at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panic.rs:133
#13 pyo3::callback::handle_panic<engine::externs::interface::__pyo3_raw_session_poll_workunits::{closure#0}, *mut pyo3::ffi::object::PyObject> (body=...)
    at /github/home/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.15.1/src/callback.rs:245
#14 engine::externs::interface::__pyo3_raw_session_poll_workunits (_slf=<optimized out>, _args=<optimized out>, _nargs=<optimized out>, _kwnames=<optimized out>) at src/externs/interface.rs:824
#15 0x000055f92471d38e in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<built-in method session_poll_workunits of module object at remote 0x7fe99eb834f0>, args=0x7fa661bee390, nargsf=<optimized out>, 
    kwnames=<optimized out>) at Objects/methodobject.c:446
#16 0x000055f92455ea39 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fa661bee390, 
    callable=<built-in method session_poll_workunits of module object at remote 0x7fe99eb834f0>, tstate=0x55f92fd065f0) at ./Include/cpython/abstract.h:118
#17 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fa661bee390, callable=<built-in method session_poll_workunits of module object at remote 0x7fe99eb834f0>)
    at ./Include/cpython/abstract.h:127
#18 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at Python/ceval.c:5075
#19 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3487
#20 0x000055f924556e9b in _PyEval_EvalFrame (throwflag=0, 
    f=Frame 0x7fa661bee200, for file $HOME/.cache/pants/setup/bootstrap-Linux-x86_64/2.9.0_py39/lib/python3.9/site-packages/pants/engine/internals/scheduler.py, line 618, in poll_workunits (self=<SchedulerSession(_scheduler=<Scheduler(include_trace_on_error=True, _visualize_to_dir=None, _visualize_run_count=0, _py_scheduler=<builtins.PyScheduler at remote 0x7fa661bd45f0>) at remote 0x7fe99c2b25e0>, _py_session=<builtins.PySession at remote 0x7fe9980a82a0>) at remote 0x7fa661ca8d30>, max_log_verbosity=<LogLevel(_value_='trace', _level=5, _name_='TRACE', __objclass__=<EnumMeta(_generate_next_value_=<function at remote 0x7fe99f36e700>, __module__='pants.util.logging', __annotations__={'_level': 'int'}, __new__=<function at remote 0x7fe99f36e670>, level=<property at remote 0x7fe99d784400>, log=<function at remote 0x7fe99d77a790>, set_level_for=<function at remote 0x7fe99d77a820>, __doc__='An enumeration.', _member_names_=['TRACE', 'DEBUG', 'INFO', 'WARN', 'ERROR'], _member_map_={'TRACE': <...>, 'DEBUG': <LogLe...(truncated), tstate=0x55f92fd065f0) at ./Include/internal/pycore_ceval.h:40
#21 function_code_fastcall (tstate=0x55f92fd065f0, co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>) at Objects/call.c:330
#22 0x000055f92455e194 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fa661bec950, callable=<function at remote 0x7fe99d20d670>, tstate=0x55f92fd065f0)
    at ./Include/cpython/abstract.h:118
#23 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fa661bec950, callable=<function at remote 0x7fe99d20d670>) at ./Include/cpython/abstract.h:127
#24 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at Python/ceval.c:5075
#25 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3504
#26 0x000055f92461fb4a in _PyEval_EvalFrame (throwflag=0, 
    f=Frame 0x7fa661bec7c0, for file $HOME/.cache/pants/setup/bootstrap-Linux-x86_64/2.9.0_py39/lib/python3.9/site-packages/pants/engine/streaming_workunit_handler.py, line 250, in poll_workunits (self=<_InnerHandler(_target=None, _name='Thread-1', _args=(), _kwargs={}, _daemonic=True, _ident=140641297385024, _native_id=3724751, _tstate_lock=<_thread.lock at remote 0x7fa661d77a50>, _started=<Event(_cond=<Condition(_lock=<_thread.lock at remote 0x7fa661c9db40>, acquire=<built-in method acquire of _thread.lock object at remote 0x7fa661c9db40>, release=<built-in method release of _thread.lock object at remote 0x7fa661c9db40>, _waiters=<collections.deque at remote 0x7fa661d07fa0>) at remote 0x7fa661c9db80>, _flag=True) at remote 0x7fa661c9dbe0>, _is_stopped=False, _initialized=True, _stderr=<builtins.PyStdioWrite at remote 0x7fe99cc24a30>, _invoke_excepthook=<function at remote 0x7fa661d1a820>, scheduler=<SchedulerSession(_scheduler=<Scheduler(include_trace_on_error=True, _visualize_to_dir=None, _visualize_run...(truncated), tstate=0x55f92fd065f0) at ./Include/internal/pycore_ceval.h:40

Your operating system and version

Ubuntu 21.04, Arch with kernel 5.13.13-arch1-1

Your Python version (python --version)

3.7, 3.9

Your Rust version (rustc --version)

1.57.0

Your PyO3 version

0.15.1

How did you install python? Did you use a virtualenv?

Inside a virtualenv, but retrieved via system-relevant package managers.

Additional Info

No response

[davidhewitt]

Thanks for reporting this. TLDR; my guess is that there's a foreign (C++?) exception which is interacting with catch_unwind and causing the crash. That's the same assumption I had when implementing #1990. I could be wrong. More investigation welcome.

My current understanding of unwinding and panics (and foreign exceptions) is derived from RFC 2945. Basically, we're stuck between a rock and a hard place, because the current state of unwinding in Rust is the following:

• It is UB for a panic! to unwind into non-Rust stack frames. This is why we need catch_unwind to avoid this UB.
• It is however unfortunately also UB for non-Rust exceptions to unwind into Rust code, and for catch_unwind to catch these unwinds.

I would be tempted to guess that pants has code throwing a C++ exception (or possibly reaches functions like pthread_exit, which implement what the RFC calls "forced unwinding"), thus reacing UB inside catch_unwind. Seems like the current state of the world is that this triggers an abort.

Unfortunately, I don't know if there's anything which PyO3 can do right now to resolve. I think possible action points for the future:

• We need to understand whether it would be correct for us to make #[pyfunction] use the proposed extern "C-unwind" ABI. I think this depends on whether the Python interpreter is compiled with exception support? This is reaching the limits of my knowledge; expert opinions welcome.
  If we can use C-unwind, then we can consider removing catch_unwind which would remove this interaction.
• Alternatively, RFC 2945 appears to leave interaction of catch_unwind with foreign exceptions unspecified. If someone was willing to author an RFC to define the interaction, this may resolve the abort (or at least make us comfortable in the knowledge this is well specified).

That said, I'm happy to explore suggestions for anything we could do in the meanwhile.

We could, for example, add a no_catch_unwind attribute or configuration to remove catch_unwind from specific code paths. That said, I'm unsure this is desirable - it's probably better to remove the UB of panics entering foreign code than it is to try to avoid a very exotic abort?

[davidhewitt]

cc @acfoltzer @BatmanAoD - in case you're interested in the interaction of catch_unwind here, or willing to offer suggestions about how PyO3 might want to handle unwinding in the future.

[stuhood]

Thanks for reporting this. TLDR; my guess is that there's a foreign (C++?) exception which is interacting with catch_unwind and causing the crash. That's the same assumption I had when implementing #1990. I could be wrong. More investigation welcome.

This could be possible, yea.

One strange datapoint against this being a (C++) exception though is that when we apply this patch: pantsbuild/pants#14253 ... to attempt to capture more details, the error no longer reproduces. The user used the branch for a few days without experiencing the issue again.

I'm going to end up landing the debug output as a "fix" for the issue/heisenbug, and will report back if shifting the signature really does end up avoiding the issue on a longer time frame.

[BatmanAoD]

This PR should ensure that catch_unwind does not catch C++ exceptions or forced-unwinding. @Amanieu I can't remember; was an abort for foreign exceptions in catch_unwind introduced after that?

[BatmanAoD]

...ah. Indeed it was: rust-lang/rust#70212

From the description, it does sound to me like removing the catch_unwind is the right long-term strategy, since the intent is merely to avoid UB. But this does seem like a compelling use-case: suppose you want to let C++ exceptions "cross through" a Rust function, but you don't want to let Rust panics escape. I think this is impossible in pure Rust (i.e. without writing C++ shims with try guards) without specifying how catch_unwind handles foreign unwinds.

[Amanieu]

From the backtrace I can tell that this is due to catch_unwind catching a forced unwind from glibc, most likely from pthread_exit. Calling pthread_exit while there are still Rust objects on the stack is UB.