Segmentation fault in sentry profiler

[IamAbbey]

How do you use Sentry?

Sentry Saas (sentry.io)

Version

1.30.0

Steps to Reproduce

The issue is hard to reproduce exactly as it occur sparingly.

The issue occurs in our Django application and whenever it happens there is a server crash with:
Example of an extract from log: Worker (pid:33) was sent SIGSEGV!

The server start command is gunicorn -b 0.0.0.0:8000 --workers 2 --threads 4 --keep-alive 5 config.wsgi --log-level info

From the logs attached below, it appears the server crashes when at a point sentry profiling resulted in Segmentation fault

Python Version: Python 3.11.5

Sentry SDK settings:

sentry_sdk.init(
    dsn=get_env_value("SENTRY_DSN"),
    # captures 100% of transactions for performance monitoring.
    # but we make further decisions in `before_send_transaction`
    traces_sample_rate=1.0,
    # profile 100% of sampled transactions.
    profiles_sample_rate=1.0,
    before_send_transaction=sentry_before_send_transaction,
    integrations=[DjangoIntegration()],
    # Associate django.contrib.auth users to errors
    send_default_pii=True,
    environment=get_env_value("SENTRY_ENVIRONMENT"),
)

Expected Result

The sentry-sdk being able to do the expected profiling correctly and reporting them to Sentry.

Actual Result

Below are the logs from analysing the coredump with gdb

(gdb) py-bt
Traceback (most recent call first):
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/profiler.py", line 273, in extract_stack
    f_back = raw_frame.f_back
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/profiler.py", line 832, in <listcomp>
    (str(tid), extract_stack(frame, cache, cwd))
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/profiler.py", line 831, in _sample_stack
    sample = [
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/profiler.py", line 931, in run
    self.sampler()
  File "/usr/local/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/threading.py", line 70, in run
    return old_run_func(self, *a, **kw)
  File "/usr/local/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.11/threading.py", line 995, in _bootstrap
    self._bootstrap_inner()

(gdb) py-list
 268        """
 269    
 270        raw_frames = deque(maxlen=max_stack_depth)  # type: Deque[FrameType]
 271    
 272        while raw_frame is not None:
>273            f_back = raw_frame.f_back
 274            raw_frames.append(raw_frame)
 275            raw_frame = f_back
 276    
 277        frame_ids = tuple(frame_id(raw_frame) for raw_frame in raw_frames)
 278        frames = []

#0  0x0000ffffa0263a8c in _PyFrame_IsIncomplete (frame=0xffff9fa9b2f8) at ./Include/internal/pycore_frame.h:151
#1  PyFrame_GetBack (frame=<optimized out>) at Objects/frameobject.c:1328
#2  0x0000ffffa027073c in frame_getback (f=<optimized out>, closure=<optimized out>) at Objects/frameobject.c:103
#3  0x0000ffffa0195730 in _PyObject_GenericGetAttrWithDict
    (obj=Frame 0xffff9fa9b378, for file /usr/local/lib/python3.11/site-packages/typing_inspect.py, line 232, in is_union_type (tp=<type at remote 0xffffa04568d8>), name='f_back', dict=0x0, suppress=0) at ./Include/object.h:133
#4  0x0000ffffa017915c in PyObject_GenericGetAttr (name='f_back', obj=Frame 0xffff9fa9b378, for file /usr/local/lib/python3.11/site-packages/typing_inspect.py, line 232, in is_union_type (tp=<type at remote 0xffffa04568d8>))
    at Objects/object.c:1368
#5  PyObject_GetAttr (v=Frame 0xffff9fa9b378, for file /usr/local/lib/python3.11/site-packages/typing_inspect.py, line 232, in is_union_type (tp=<type at remote 0xffffa04568d8>), name='f_back') at Objects/object.c:916
#6  0x0000ffffa0182870 in _PyEval_EvalFrameDefault (tstate=0xffff90001ed0, frame=0xffff9fe18428, throwflag=-1614065056) at Python/ceval.c:3466
#7  0x0000ffffa01805d4 in _PyEval_EvalFrame (throwflag=0, frame=0xffff9fe18230, tstate=0xffff90001ed0) at ./Include/internal/pycore_ceval.h:73
#8  _PyEval_Vector (tstate=0xffff90001ed0, func=func@entry=0xffff9d1ecae0, locals=locals@entry=0x0, args=args@entry=0xffff8fffe4f8, argcount=argcount@entry=1, kwnames=kwnames@entry=0x0) at Python/ceval.c:6439
#9  0x0000ffffa01b8b1c in _PyFunction_Vectorcall (kwnames=0x0, nargsf=1, stack=0xffff8fffe4f8, func=<function at remote 0xffff9d1ecae0>) at Objects/call.c:393
#10 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0xffff8fffe4f8, callable=<function at remote 0xffff9d1ecae0>, tstate=0xffff90001ed0) at ./Include/internal/pycore_call.h:92
#11 method_vectorcall (method=<optimized out>, args=0xffffa04886f8 <_PyRuntime+58944>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/classobject.c:67
#12 0x0000ffffa0184414 in _PyEval_EvalFrameDefault (tstate=0xffff90001ed0, frame=0xffff9fe181b8, throwflag=-1614065056) at Python/ceval.c:5381
#13 0x0000ffffa01805d4 in _PyEval_EvalFrame (throwflag=0, frame=0xffff9fe181b8, tstate=0xffff90001ed0) at ./Include/internal/pycore_ceval.h:73
#14 _PyEval_Vector (tstate=0xffff90001ed0, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6439
#15 0x0000ffffa0184414 in _PyEval_EvalFrameDefault (tstate=0xffff90001ed0, frame=0xffff9fe18110, throwflag=-1614065056) at Python/ceval.c:5381
#16 0x0000ffffa01805d4 in _PyEval_EvalFrame (throwflag=0, frame=0xffff9fe18020, tstate=0xffff90001ed0) at ./Include/internal/pycore_ceval.h:73
#17 _PyEval_Vector (tstate=0xffff90001ed0, func=func@entry=0xffff9f68a020, locals=locals@entry=0x0, args=args@entry=0xffff8fffe8c8, argcount=argcount@entry=1, kwnames=kwnames@entry=0x0) at Python/ceval.c:6439
#18 0x0000ffffa01b8b1c in _PyFunction_Vectorcall (kwnames=0x0, nargsf=1, stack=0xffff8fffe8c8, func=<function at remote 0xffff9f68a020>) at Objects/call.c:393
#19 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0xffff8fffe8c8, callable=<function at remote 0xffff9f68a020>, tstate=0xffff90001ed0) at ./Include/internal/pycore_call.h:92
#20 method_vectorcall (method=<optimized out>, args=0xffffa04886f8 <_PyRuntime+58944>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/classobject.c:67
#21 0x0000ffffa026b230 in thread_run (boot_raw=0xffff95890f90) at ./Modules/_threadmodule.c:1092
#22 0x0000ffffa026b1d0 in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:241
#23 0x0000ffff9fe9ee18 in start_thread (arg=0xffff9555d537) at ./nptl/pthread_create.c:442
#24 0x0000ffff9ff07e9c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

[Zylphrex]

Thanks for the report. This looks like a problem with python but we'll take a look to see if there's anything we can do to mitigate it from our end.

[zyv]

@Zylphrex it would be great if you could open a CPython issue and mention us if you conclude that there is nothing you are doing wrong at your end. I've pinged someone in python/cpython#99110 who worked on similar issues in CPython 3.11 / 3.12, but so far got no response.

[Zylphrex]

@zyv I haven't been able to identify anything that we can do from the SDK to avoid the segfault without disabling the profiler entirely.

Also, it would be great if you could provide some more info about your environment that may be helpful in reproducing this problem.

1. output of python -VV
2. your gunicorn config
3. output of pip freeze

Also some questions:

1. Does this happen on app startup? Or does it happen after some time?
2. Was an interrupt sent to the app?

[catcombo]

output of python -VV

Python 3.11.5 (main, Sep 7 2023, 18:53:16) [GCC 12.2.0]

your gunicorn config

gunicorn -b 0.0.0.0:8000 --workers 2 --threads 4 --keep-alive 5 config.wsgi --log-file - --access-logfile - --log-level debug --enable-stdio-inheritance

output of pip freeze

add-trailing-comma==3.0.1
amqp==5.1.1
annotated-types==0.5.0
asgiref==3.7.2
attrs==23.1.0
bcrypt==4.0.1
beautifulsoup4==4.12.2
billiard==4.1.0
black==23.7.0
boto3==1.28.37
botocore==1.31.37
Brotli==1.0.9
build==0.10.0
CacheControl==0.13.1
cachetools==5.3.1
celery==5.3.1
certifi==2023.7.22
cffi==1.15.1
charset-normalizer==3.2.0
cleo==2.0.1
click==8.1.7
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.3.0
coverage==7.3.0
crashtest==0.4.1
cryptography==41.0.3
cssselect2==0.7.0
dataclasses-json==0.5.14
Deprecated==1.2.14
distlib==0.3.7
dj-database-url==2.1.0
Django==4.2.4
django-bootstrap-v5==1.0.11
django-cid==2.3
django-cors-headers==4.2.0
django-countries==7.5.1
django-debug-toolbar==4.2.0
django-oauth-toolkit==2.3.0
djangorestframework==3.14.0
dulwich==0.21.6
et-xmlfile==1.1.0
filelock==3.12.3
flake8==6.1.0
flake8-black==0.3.6
flake8-coding==1.3.2
flake8-isort==6.0.0
flake8-quotes==3.3.2
fonttools==4.42.1
gunicorn==21.2.0
html5lib==1.1
idna==3.4
importlib-metadata==6.8.0
iniconfig==2.0.0
installer==0.7.0
isort==5.12.0
jaraco.classes==3.3.0
jeepney==0.8.0
Jinja2==3.1.2
jmespath==1.0.1
jsonfield==3.1.0
jsonschema==4.17.3
jwcrypto==1.5.0
keyring==24.2.0
kombu==5.3.1
lxml==4.9.3
Markdown==3.4.4
MarkupSafe==2.1.3
marshmallow==3.20.1
mccabe==0.7.0
model-bakery==1.15.0
more-itertools==10.1.0
msgpack==1.0.5
mypy-extensions==1.0.0
oauthlib==3.2.2
openpyxl==3.1.2
oyaml==1.0
packaging==23.1
paramiko==3.3.1
pathspec==0.11.2
pexpect==4.8.0
Pillow==10.0.0
pkginfo==1.9.6
platformdirs==3.10.0
pluggy==1.3.0
poetry==1.6.1
poetry-core==1.7.0
poetry-plugin-export==1.5.0
prompt-toolkit==3.0.39
psycopg==3.1.10
psycopg-binary==3.1.10
ptyprocess==0.7.0
pycodestyle==2.11.0
pycparser==2.21
pydantic==2.3.0
pydantic_core==2.6.3
pydyf==0.7.0
pyflakes==3.1.0
pyhumps==3.8.0
PyJWT==2.8.0
PyNaCl==1.5.0
pyphen==0.14.0
pyproject_hooks==1.0.0
pyrsistent==0.19.3
PySocks==1.7.1
pytest==7.4.0
pytest-cov==4.1.0
pytest-django==4.5.2
python-dateutil==2.8.2
pytz==2023.3
PyYAML==6.0.1
rapidfuzz==2.15.1
redis==5.0.0
requests==2.31.0
requests-mock==1.11.0
requests-oauthlib==1.3.1
requests-toolbelt==1.0.0
s3transfer==0.6.2
SecretStorage==3.3.3
sentry-sdk==1.30.0
shellingham==1.5.3
six==1.16.0
soupsieve==2.4.1
sqlparse==0.4.4
tinycss2==1.2.1
tokenize-rt==5.2.0
tomlkit==0.12.1
tqdm==4.66.1
trove-classifiers==2023.8.7
typing-inspect==0.9.0
typing_extensions==4.7.1
tzdata==2023.3
urllib3==1.26.16
vine==5.0.0
virtualenv==20.24.5
watchtower==3.0.1
wcwidth==0.2.6
weasyprint==59.0
webencodings==0.5.1
whitenoise==6.5.0
wrapt==1.15.0
youtrack-sdk @ git+ssh://git@github.com/moneymeets/youtrack-sdk.git@4071e614e997be70c219abc6d21e1c33d759f5a6
zipp==3.16.2
zopfli==0.2.2