Segfault in test_rejects_different_header_magic

[stefanor]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

From https://bugs.debian.org/1122076:

tests/unit/test_reader.py::test_rejects_different_header_magic is (intermittently) segfaulting under Python 3.14.

On 64c0e81:

$ PYTHONPATH=$(pwd)/src ve314/bin/python -m pytest tests/unit/test_reader.py -k test_rejects_different_header_magic
========================================= test session starts ==========================================
platform linux -- Python 3.14.2, pytest-8.4.2, pluggy-1.6.0
rootdir: /tmp/memray
configfile: pyproject.toml
plugins: syrupy-4.8.0, cov-7.0.0, textual-snapshot-1.1.0
collected 5 items / 4 deselected / 1 selected                                                          

tests/unit/test_reader.py .                                                                      [100%]

=================================== 1 passed, 4 deselected in 0.04s ====================================
$ PYTHONPATH=$(pwd)/src ve314/bin/python -m pytest tests/unit/test_reader.py -k test_rejects_different_header_magic
========================================= test session starts ==========================================
platform linux -- Python 3.14.2, pytest-8.4.2, pluggy-1.6.0
rootdir: /tmp/memray
configfile: pyproject.toml
plugins: syrupy-4.8.0, cov-7.0.0, textual-snapshot-1.1.0
collected 5 items / 4 deselected / 1 selected                                                          

tests/unit/test_reader.py Fatal Python error: Segmentation fault

Current thread 0x00007f63fc52b200 [python] (most recent call first):
  File "/tmp/memray/tests/unit/test_reader.py", line 30 in test_rejects_different_header_magic
  File "/tmp/memray/ve314/lib/python3.14/site-packages/_pytest/python.py", line 157 in pytest_pyfunc_call
  File "/tmp/memray/ve314/lib/python3.14/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/tmp/memray/ve314/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/tmp/memray/ve314/lib/python3.14/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/tmp/memray/ve314/lib/python3.14/site-packages/_pytest/python.py", line 1671 in runtest
  File "/tmp/memray/ve314/lib/python3.14/site-packages/_pytest/runner.py", line 178 in pytest_runtest_call
  File "/tmp/memray/ve314/lib/python3.14/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/tmp/memray/ve314/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/tmp/memray/ve314/lib/python3.14/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/tmp/memray/ve314/lib/python3.14/site-packages/_pytest/runner.py", line 246 in <lambda>
  File "/tmp/memray/ve314/lib/python3.14/site-packages/_pytest/runner.py", line 344 in from_call
  File "/tmp/memray/ve314/lib/python3.14/site-packages/_pytest/runner.py", line 245 in call_and_report
  File "/tmp/memray/ve314/lib/python3.14/site-packages/_pytest/runner.py", line 136 in runtestprotocol
  File "/tmp/memray/ve314/lib/python3.14/site-packages/_pytest/runner.py", line 117 in pytest_runtest_protocol
  File "/tmp/memray/ve314/lib/python3.14/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/tmp/memray/ve314/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/tmp/memray/ve314/lib/python3.14/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/tmp/memray/ve314/lib/python3.14/site-packages/_pytest/main.py", line 367 in pytest_runtestloop
  File "/tmp/memray/ve314/lib/python3.14/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/tmp/memray/ve314/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/tmp/memray/ve314/lib/python3.14/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/tmp/memray/ve314/lib/python3.14/site-packages/_pytest/main.py", line 343 in _main
  File "/tmp/memray/ve314/lib/python3.14/site-packages/_pytest/main.py", line 289 in wrap_session
  File "/tmp/memray/ve314/lib/python3.14/site-packages/_pytest/main.py", line 336 in pytest_cmdline_main
  File "/tmp/memray/ve314/lib/python3.14/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/tmp/memray/ve314/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/tmp/memray/ve314/lib/python3.14/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/tmp/memray/ve314/lib/python3.14/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/tmp/memray/ve314/lib/python3.14/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/tmp/memray/ve314/lib/python3.14/site-packages/pytest/__main__.py", line 9 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main

Current thread's C stack trace (most recent call first):
  Binary file "ve314/bin/python", at _Py_DumpStack+0x4a [0x472522]
  Binary file "ve314/bin/python" [0x4fc6c7]
  Binary file "/usr/lib/x86_64-linux-gnu/libc.so.6", at +0x40a50 [0x7f63fc240a50]
  Binary file "/usr/lib/x86_64-linux-gnu/libgcc_s.so.1", at +0x25ef0 [0x7f63fb413ef0]
  Binary file "/usr/lib/x86_64-linux-gnu/libgcc_s.so.1", at __gcc_personality_v0+0xd9 [0x7f63fb4140c9]
  Binary file "/usr/lib/x86_64-linux-gnu/libunwind.so.8", at __libunwind_Unwind_RaiseException+0x1c0 [0x7f63fb3aa740]
  Binary file "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", at __cxa_throw+0x3b [0x7f63f94b93eb]
  Binary file "/tmp/memray/src/memray/_memray.cpython-314-x86_64-linux-gnu.so", at +0x2bdeb [0x7f63f97b4deb]
  Binary file "/tmp/memray/src/memray/_memray.cpython-314-x86_64-linux-gnu.so", at +0x4cb38 [0x7f63f97d5b38]
  Binary file "/tmp/memray/src/memray/_memray.cpython-314-x86_64-linux-gnu.so", at +0x77524 [0x7f63f9800524]
  Binary file "ve314/bin/python", at _PyObject_MakeTpCall+0x241 [0x526f51]
  Binary file "ve314/bin/python", at _PyEval_EvalFrameDefault+0x3607 [0x548757]
  Binary file "ve314/bin/python" [0x583944]
  Binary file "ve314/bin/python" [0x52e8be]
  Binary file "ve314/bin/python" [0x626aff]
  Binary file "ve314/bin/python", at _PyObject_MakeTpCall+0x36c [0x52707c]
  Binary file "ve314/bin/python", at _PyEval_EvalFrameDefault+0x10a51 [0x555ba1]
  Binary file "ve314/bin/python" [0x583944]
  Binary file "ve314/bin/python" [0x52e8be]
  Binary file "ve314/bin/python" [0x626aff]
  Binary file "ve314/bin/python" [0x68a579]
  Binary file "ve314/bin/python", at _PyEval_EvalFrameDefault+0x4a2c [0x549b7c]
  Binary file "ve314/bin/python" [0x583944]
  Binary file "ve314/bin/python" [0x52e8be]
  Binary file "ve314/bin/python" [0x626aff]
  Binary file "ve314/bin/python", at _PyObject_MakeTpCall+0x36c [0x52707c]
  Binary file "ve314/bin/python", at _PyEval_EvalFrameDefault+0x10a51 [0x555ba1]
  Binary file "ve314/bin/python" [0x583944]
  Binary file "ve314/bin/python" [0x52e8be]
  Binary file "ve314/bin/python" [0x626aff]
  Binary file "ve314/bin/python", at _PyObject_MakeTpCall+0x36c [0x52707c]
  Binary file "ve314/bin/python", at _PyEval_EvalFrameDefault+0x10a51 [0x555ba1]
  <truncated rest of calls>

Extension modules: markupsafe._speedups, memray._memray, memray._test_utils (total: 3)
Segmentation fault         PYTHONPATH=$(pwd)/src ve314/bin/python -m pytest tests/unit/test_reader.py -k test_rejects_different_header_magic

Expected Behavior

No segfaults is nice.

Steps To Reproduce

1. On debian unstable
2. apt install git python3.14-dev python3.14-venv
3. git clone memray
4. python3.14 -m venv ve314`
5. ve314/bin/python -m pip install -r requirements-test.txt
6. make build-ext PYTHON=ve314/bin/python
7. PYTHONPATH=$(pwd)/src ve314/bin/python -m pytest tests/unit/test_reader.py -k test_rejects_different_header_magic

Memray Version

git HEAD

Python Version

3.14

Operating System

Linux

Anything else?

No response

[pablogsal]

Seems that Debian's Python is doing something weird here. The crash seems to be because there is symbol interposition between libunwind and libgcc_s:

1. memray links -lunwind for stack tracing
2. libunwind exports _Unwind_RaiseException (unversioned symbol at 0x5580)
3. libgcc_s exports _Unwind_RaiseException@@GCC_3.0 (versioned symbol)
4. Due to library load order, when C++ throws an exception (std::ios_base::failure), libunwind's implementation is called instead of libgcc_s's
5. libunwind's _Unwind_RaiseException doesn't properly handle the GCC C++ exception personality function (__gcc_personality_v0), causing a crash

Stack trace

  #0  libgcc_s.so.1 (crash/abort in personality handler)
  #1  __gcc_personality_v0 (libgcc_s.so.1)
  #2  __libunwind_Unwind_RaiseException (libunwind.so.8)  <-- wrong unwinder!
  #3  __cxa_throw (libstdc++.so.6)
  #4  RecordReader::readHeader throwing std::ios_base::failure

[pablogsal]

I don't understand yet why this only happens on Debian but it happens when we throw here:

0          if (!d_input->read(header.magic, sizeof(MAGIC)) || (memcmp(header.magic, MAGIC, sizeof(MAGIC)) != 0))
71          {
72              throw std::ios_base::failure(
73                      "The provided input file does not look like a binary generated by memray.");
74          }

Looks like the version of libunwind that debian unstable is using has problem's with libgcc's libunwind unwinder

[pablogsal]

The more I look into this the more I am sure this is something weird between libgcc and libunwind. We are just linking against those and raising an exception and we cannot reproduce this in any other place outside debian.

[pablogsal]

Look at this, somehow libunwind ends overriding the PLT of the libgcc unwinder:

Thread 1 "python" hit Breakpoint 1, 0x00007f345d49a3b0 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) disas
Dump of assembler code for function __cxa_throw:
=> 0x00007f345d49a3b0 <+0>:     endbr64
   0x00007f345d49a3b4 <+4>:     push   %r12
   0x00007f345d49a3b6 <+6>:     mov    %rdx,%r12
   0x00007f345d49a3b9 <+9>:     push   %rbp
   0x00007f345d49a3ba <+10>:    mov    %rsi,%rbp
   0x00007f345d49a3bd <+13>:    push   %rbx
   0x00007f345d49a3be <+14>:    mov    %rdi,%rbx
   0x00007f345d49a3c1 <+17>:    nop
   0x00007f345d49a3c2 <+18>:    call   0x7f345d485a90 <__cxa_get_globals@plt>
   0x00007f345d49a3c7 <+23>:    mov    %r12,%rdx
   0x00007f345d49a3ca <+26>:    mov    %rbp,%rsi
   0x00007f345d49a3cd <+29>:    mov    %rbx,%rdi
   0x00007f345d49a3d0 <+32>:    addl   $0x1,0x8(%rax)
   0x00007f345d49a3d4 <+36>:    call   0x7f345d485b00 <__cxa_init_primary_exception@plt>
   0x00007f345d49a3d9 <+41>:    movl   $0x1,(%rax)
   0x00007f345d49a3df <+47>:    lea    0x60(%rax),%rbx
   0x00007f345d49a3e3 <+51>:    mov    %rbx,%rdi
   0x00007f345d49a3e6 <+54>:    call   0x7f345d484ac0 <_Unwind_RaiseException@plt>
   0x00007f345d49a3eb <+59>:    mov    %rbx,%rdi
   0x00007f345d49a3ee <+62>:    call   0x7f345d483590 <__cxa_begin_catch@plt>
   0x00007f345d49a3f3 <+67>:    call   0x7f345d487260 <_ZSt9terminatev@plt>
End of assembler dump.
(gdb) disas 0x7f345d484ac0
Dump of assembler code for function _Unwind_RaiseException@plt:
   0x00007f345d484ac0 <+0>:     jmp    *0x1cc282(%rip)        # 0x7f345d650d48 <_Unwind_RaiseException@got.plt>
   0x00007f345d484ac6 <+6>:     push   $0x1a9
   0x00007f345d484acb <+11>:    jmp    0x7f345d483020
End of assembler dump.
(gdb) x/gx 0x7f345d650d48
0x7f345d650d48 <_Unwind_RaiseException@got.plt>:        0x00007f345f221580
(gdb) disas 0x00007f345f221580
Dump of assembler code for function __libunwind_Unwind_RaiseException:
   0x00007f345f221580 <+0>:     push   %r15
   0x00007f345f221582 <+2>:     push   %r14
   0x00007f345f221584 <+4>:     push   %r13
   0x00007f345f221586 <+6>:     push   %r12
   0x00007f345f221588 <+8>:     push   %rbp
   0x00007f345f221589 <+9>:     mov    %rdi,%rbp
   0x00007f345f22158c <+12>:    push   %rbx

[pablogsal]

Yes this is a libunwind packaging bug in debian:

  1. libunwind exports _Unwind_RaiseException (Base version, no symbol versioning)
  2. libgcc_s exports _Unwind_RaiseException@@GCC_3.0 (properly versioned)
  3. When memray loads, libunwind loads first → its unversioned symbol intercepts libstdc++'s versioned lookup
  4. libunwind's _Unwind_RaiseException is incompatible with GCC's __gcc_personality_v0
  5. Crash happens during C++ exception unwinding

as evidence, if you do LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgcc_s.so.1 before running your reproducer it works.

Indeed, libunwind should NOT be exporting _Unwind_RaiseException unversioned! In my system it clearly doesn't:

$ nm -D /lib/libunwind.so.8 | grep _Unwind_RaiseException
(empty)

but in Debian it does:

$ nm -D /usr/lib/x86_64-linux-gnu/libunwind.so.8 | grep _Unwind_RaiseException
0000000000005580 T _Unwind_RaiseException
0000000000005580 T __libunwind_Unwind_RaiseException

although clearly there is something else at play because old Debian also exports this without crashing...

[pablogsal]

Seems related: https://reviews.llvm.org/D106703

[pablogsal]

The linker bindings:

       852:     symbol=_Unwind_RaiseException;  lookup in file=python [0]
       852:     symbol=_Unwind_RaiseException;  lookup in file=/usr/lib/x86_64-linux-gnu/libm.so.6 [0]
       852:     symbol=_Unwind_RaiseException;  lookup in file=/usr/lib/x86_64-linux-gnu/libz.so.1 [0]
       852:     symbol=_Unwind_RaiseException;  lookup in file=/usr/lib/x86_64-linux-gnu/libexpat.so.1 [0]
       852:     symbol=_Unwind_RaiseException;  lookup in file=/usr/lib/x86_64-linux-gnu/libc.so.6 [0]
       852:     symbol=_Unwind_RaiseException;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
       852:     symbol=_Unwind_RaiseException;  lookup in file=/memray/src/memray/_memray.cpython-314-x86_64-linux-gnu.so [0]
       852:     symbol=_Unwind_RaiseException;  lookup in file=/usr/lib/x86_64-linux-gnu/liblz4.so.1 [0]
       852:     symbol=_Unwind_RaiseException;  lookup in file=/usr/lib/x86_64-linux-gnu/libunwind.so.8 [0]
       852:     binding file /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 [0] to /usr/lib/x86_64-linux-gnu/libunwind.so.8 [0]: normal
symbol `_Unwind_RaiseException' [GCC_3.0]

so when _Unwind_RaiseException is requested the linker goes and uses the version in libunwind that somehow is invalid.

[pablogsal]

@godlygeek we can hack-fix it by forcing memray to link against libgcc_s.so.1 first before libunwind but this feels like a dirty hack

[pablogsal]

@stefanor please check with the libunwind and gcc maintainers because Debian here is doing something very weird.

[pablogsal]

@stefanor here is a reproducer without memray. With this script:

#!/bin/bash
# Reproducer for libunwind + Python C++ exception crash
# Run inside: docker run --rm -it debian:unstable bash
# Then: bash reproduce.sh

set -e

apt-get update -qq
apt-get install -y -qq build-essential python3.14-dev python3.14-venv libunwind-dev

cd /tmp
mkdir -p repro && cd repro

cat > repro.cpp << 'EOF'
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include <stdexcept>

#define UNW_LOCAL_ONLY
#include <libunwind.h>

extern void do_throw();

__attribute__((constructor))
static void init() {
    unw_set_caching_policy(unw_local_addr_space, UNW_CACHE_PER_THREAD);
}

static PyObject* trigger(PyObject* self, PyObject* args) {
    try {
        do_throw();
    } catch (...) {
        PyErr_SetString(PyExc_OSError, "caught");
        return nullptr;
    }
    Py_RETURN_NONE;
}

static PyMethodDef methods[] = {{"trigger", trigger, METH_NOARGS, ""}, {nullptr}};
static struct PyModuleDef module = {PyModuleDef_HEAD_INIT, "repro", nullptr, -1, methods};
PyMODINIT_FUNC PyInit_repro(void) { return PyModule_Create(&module); }
EOF

cat > thrower.cpp << 'EOF'
#include <stdexcept>

__attribute__((noinline))
void do_throw() {
    throw std::runtime_error("test");
}
EOF

cat > setup.py << 'EOF'
from setuptools import setup, Extension
setup(name="repro", ext_modules=[Extension(
    "repro", ["repro.cpp", "thrower.cpp"], language="c++",
    extra_compile_args=["-std=c++17", "-flto", "-fvisibility=hidden"],
    extra_link_args=["-lunwind", "-flto"]
)])
EOF

cat > test_repro.py << 'EOF'
import pytest, repro

def test_it():
    with pytest.raises(OSError):
        repro.trigger()
EOF

python3.14 -m venv venv
source venv/bin/activate
pip install -q setuptools pytest

python setup.py build_ext --inplace

echo "=== Library dependencies ==="
ldd repro*.so

echo "=== Direct call (works) ==="
python -c 'import repro; repro.trigger()' 2>&1 && echo "No crash" || echo "Crashed"

echo "=== Pytest (crashes) ==="
python -m pytest test_repro.py -x

if you run:

cat reproduce.sh | docker run --rm -i debian:unstable /bin/bash

it segfaults

[stefanor]

@pablogsal: Thanks for digging into that. @doko42 @AdrianBunk

[pablogsal]

Ok, I found more apparently is faulthandler what triggers the problem (because it links against libgcc_s - still unclear why only this is a problem on debian). Here is a much smaller reproducer

#!/bin/bash
# Minimal reproducer: libunwind + faulthandler symbol conflict
# Run: cat reproduce.sh | docker run --rm -i debian:unstable /bin/bash

set -e
apt-get update -qq
apt-get install -y -qq build-essential python3.14-dev libunwind-dev

cd /tmp && mkdir -p repro && cd repro

# Extension that uses libunwind and throws C++ exceptions
cat > repro.cpp << 'EOF'
#include <Python.h>
#define UNW_LOCAL_ONLY
#include <libunwind.h>
#include <stdexcept>

extern void do_throw();

__attribute__((constructor))
static void init() { unw_set_caching_policy(unw_local_addr_space, UNW_CACHE_PER_THREAD); }

static PyObject* trigger(PyObject* self, PyObject* args) {
    try { do_throw(); } catch (...) { Py_RETURN_NONE; }
    Py_RETURN_NONE;
}

static PyMethodDef methods[] = {{"trigger", trigger, METH_NOARGS, NULL}, {NULL}};
static struct PyModuleDef module = {PyModuleDef_HEAD_INIT, "repro", NULL, -1, methods};
PyMODINIT_FUNC PyInit_repro(void) { return PyModule_Create(&module); }
EOF

cat > thrower.cpp << 'EOF'
#include <stdexcept>
void do_throw() { throw std::runtime_error("x"); }
EOF

g++ -shared -fPIC -flto -fvisibility=hidden \
    $(python3.14-config --includes) \
    repro.cpp thrower.cpp -lunwind -o repro.so

echo "=== WORKS: import repro first ==="
python3.14 -c 'import repro; import faulthandler; faulthandler.enable(); repro.trigger(); print("OK")'

echo "=== CRASHES: faulthandler.enable() first ==="
python3.14 -c 'import faulthandler; faulthandler.enable(); import repro; repro.trigger(); print("OK")'

[pablogsal]

The chain:

1. faulthandler.enable() calls glibc's backtrace()
2. backtrace() loads libgcc_s for stack unwinding
3. libgcc_s binds _Unwind_RaiseException to itself (libunwind not loaded yet)
4. Later when repro.so loads libunwind, libstdc++ binds to libunwind
5. Mismatch → crash

[stefanor]

Sounds like I put this bug in front of exactly the right person to look into it, thanks! (You really do attract all the linker bugs) It would have taken me a lot longer to figure out.

I see the mismatch in the chain you describe, but I also don't see anything doing anything obviously wrong. backtrace() and libunwind seem to just be incompatible. Is there a reasonable path where libunwind takes over _Unwind_RaiseException and both faulthandler and libunwind work?