Pypy 3.9 segfaults while rewriting test assertions

[LilyFirefly]

• a detailed description of the bug or problem you are having

When testing the project I work on in CI I frequently see a segfault in pytest's test assert rewriting when running with pypy 3.9. (I haven't yet seen it with 3.10, but I only just added that to CI.)

Run coverage run -m pytest --ignore=tests/simplepoll
  coverage run -m pytest --ignore=tests/simplepoll
  shell: /usr/bin/bash -e {0}
  env:
    CARGO_INCREMENTAL: 0
    CARGO_TERM_COLOR: always
    CACHE_ON_FAILURE: false
    pythonLocation: /opt/hostedtoolcache/PyPy/3.9.16/x64/bin
    PYTHONFAULTHANDLER: 1
    RUST_BACKTRACE: 1
Fatal Python error: Segmentation fault

Stack (most recent call first, approximate line numbers):
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/assertion/rewrite.py", line 346 in _rewrite_test
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/assertion/rewrite.py", line 138 in exec_module
  File "<frozen importlib._bootstrap>", line 659 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1002 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1018 in _gcd_import
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/importlib/__init__.py", line 109 in import_module
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/pathlib.py", line 486 in import_path
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 613 in _importtestmodule
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 527 in _getobj
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 305 in obj
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 536 in _inject_setup_module_fixture
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 530 in collect
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 372 in <lambda>
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 318 in from_call
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 371 in pytest_make_collect_report
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 30 in _multicall
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 103 in _hookexec
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 427 in __call__
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 544 in collect_one_node
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 827 in genitems
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 827 in genitems
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 610 in perform_collect
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 333 in pytest_collection
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 30 in _multicall
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 103 in _hookexec
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 427 in __call__
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 320 in _main
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 257 in wrap_session
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 30 in _multicall
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 103 in _hookexec
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 427 in __call__
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 134 in main
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 182 in console_main
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/pytest/__main__.py", line 1 in <module>
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/coverage/execfile.py", line 169 in run
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/coverage/cmdline.py", line 813 in do_run
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/coverage/cmdline.py", line 611 in command_line
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/lib/pypy3.9/site-packages/coverage/cmdline.py", line 964 in main
  File "/opt/hostedtoolcache/PyPy/3.9.16/x64/bin/coverage", line 3 in <module>
  File "<builtin>/app_main.py", line 1012 in execfile
  File "<builtin>/app_main.py", line 131 in run_toplevel
  File "<builtin>/app_main.py", line 733 in run_command_line
  File "<builtin>/app_main.py", line 1126 in entry_point
/home/runner/actions-runner/_work/_temp/4b79962b-6f8c-4a60-9e1a-65886acc7eb7.sh: line 1:  1738 Segmentation fault      (core dumped) coverage run -m pytest --ignore=tests/simplepoll
Error: Process completed with exit code 139.

• output of pip list from the virtual environment you are using

Package                Version               Editable project location
---------------------- --------------------- --------------------------------------------------
amqp                   5.1.1
anyio                  3.7.0
asgiref                3.6.0
attrs                  23.1.0
billiard               4.1.0
black                  23.3.0
boto3                  1.27.0
botocore               1.30.0
build                  0.10.0
celery                 5.3.1
Cerberus               1.3.4
certifi                2023.5.7
cffi                   1.15.1
charset-normalizer     3.1.0
click                  8.1.3
click-didyoumean       0.3.0
click-plugins          1.1.1
click-repl             0.3.0
colored                1.4.4
coverage               7.2.7
dj-database-url        2.0.0
Django                 3.2.20
django-environ         0.10.0
django-hashid-field    3.3.7
django-pgcrypto-fields 2.6.0
djangorestframework    3.14.0
exceptiongroup         1.1.2
factory-boy            3.2.1
Faker                  18.11.2
freezegun              1.2.2
greenlet               0.4.13
h11                    0.14.0
hashids                1.3.1
hpy                    0.0.4.dev179+g9b5d200
httpcore               0.17.2
httpretty              1.1.4
httpx                  0.24.1
huey                   2.4.5
idna                   3.4
iniconfig              2.0.0
Jinja2                 3.1.2
jmespath               1.0.1
kolo                   2.11.0+local          /home/runner/actions-runner/_work/kolo/kolo/python
kombu                  5.3.1
MarkupSafe             2.1.3
maturin                1.1.0
more-itertools         9.1.0
mypy-extensions        1.0.0
packaging              23.1
pathspec               0.11.1
pip                    23.0.1
platformdirs           3.8.0
pluggy                 1.2.0
prompt-toolkit         3.0.38
pyproject_hooks        1.0.0
pytest                 7.4.0
pytest-asyncio         0.21.0
pytest-celery          0.0.0
pytest-django          4.5.2
pytest-httpx           0.22.0
python-dateutil        2.8.2
pytz                   2023.3
readline               6.2.4.1
requests               2.31.0
s3transfer             0.6.1
setuptools             58.1.0
six                    1.16.0
sniffio                1.3.0
sqlglot                17.0.0
sqlparse               0.4.4
syrupy                 4.0.4
temppathlib            1.2.0
tomli                  2.0.1
tomli_w                1.0.0
toolz                  0.12.0
types-freezegun        1.1.10
types-requests         2.31.0.1
types-urllib3          1.26.25.13
typing_extensions      4.7.1
tzdata                 2023.3
ulid-py                1.1.0
urllib3                1.26.15
vine                   5.0.0
wcwidth                0.2.6

• pytest and operating system versions
  pytest 7.4.0
  ubuntu 22.04 (buildjet-2vcpu-ubuntu-2204 https://buildjet.com/for-github-actions/docs/runners/hardware)

• minimal example if possible
  I'm not sure how to narrow down to the problematic files.

[Zac-HD]

Thanks for the report! I think this is probably a PyPy bug, but let's see if we can narrow it down a bit before we report it upstream.

Since it seems to happen while rewriting assertions, I expect it's possible to trigger this with a single test file, and there's good tooling to analyse what's happening from there.

• Do you have a reproducible example? The goal here is a local shell command which reliably triggers the segfault, even if we have to run pytest many times in a loop - debugging when you have to wait for CI each time is... not fun, and being unable to trust your observations is even worse.
• Let's try to narrow down which files are implicated. Can you run pytest on (roughly) half of your test suite, and bisect that way? Alternatively, you could patch pytest to print what module it's about to rewrite (with flush=True) and then see what's printed just before the segfault.
• Try simplifying the environment and removing dependencies. This is much easier once you have a single file; at that point try deleting chunks of code, imports, etc. with the goal of reproducing in a virtual environment with no proprietary code - that confirms it's an OSS bug somewhere, and lets us try to debug it too.

[LilyFirefly]

Sigh, I've spent the whole evening trying to replicate locally and every time the test suite has just passed without triggering the segfault. I've also tried running the github action locally using https://github.com/nektos/act and also gotten nowhere. I'm not sure where to go from here. 😞

[Zac-HD]

Thank you for trying! This is the worst kind of bug, to be clear, HypothesisWorks/hypothesis#3028 got us stuck for several months before we actually worked it out.

Maybe it's dependent on something about the virtualization setup, or on the hardware architecture of your CI runners? Do you know what the physical CPU is on those machines?

[The-Compiler]

Perhaps trying to get something like gdb -ex "set debuginfod enabled on" -ex r -ex bt -ex q --args $(which python) -m pytest to run on GitHub Actions could help, since it hopefully will end up displaying a C stacktrace.

[LilyFirefly]

I can now confirm I see this sometimes with pypy 3.10.

@Zac-HD I think the physical cpu is AMD, but I don't know specifics beyond that.

I have also added the gdb command @The-Compiler recommended and I got this:

Run gdb -ex "set debuginfod enabled on" -ex r -ex bt -ex q --args $(which python) -m coverage run -m pytest --ignore=tests/simplepoll
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/python...
Reading symbols from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/pypy3.9.debug...
(No debugging symbols found in /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/pypy3.9.debug)
Starting program: /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/python -m coverage run -m pytest --ignore=tests/simplepoll
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Detaching after fork from child process 1955]
[Detaching after fork from child process 1956]

Program received signal SIGBUS, Bus error.
0x00007ffff5a7f9ea in pypy_g_from_object_6 () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#0  0x00007ffff5a7f9ea in pypy_g_from_object_6 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#1  0x00007ffff5a9e84d in pypy_g_from_object_59 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#2  0x00007ffff5a800d0 in pypy_g_from_object_6 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#3  0x00007ffff5a81aa0 in pypy_g_from_object_4.part ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#4  0x00007ffff5a8238e in pypy_g_from_object ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#5  0x00007ffff5bc0374 in pypy_g_compile ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#6  0x00007ffff591b050 in pypy_g_BuiltinActivation_UwS_ObjSpace_W_Root_fsencode_t () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#249 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#250 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#251 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#252 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#253 0x00007ffff632538b in pypy_g_CALL_METHOD_KW__AccessDirect_star_1 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#254 0x00007ffff5a2e7c5 in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#255 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#256 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#257 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#258 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#259 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#260 0x00007ffff5a2b2bf in pypy_g_CALL_FUNCTION__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#261 0x00007ffff5a30e57 in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#262 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#263 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#264 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#265 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#266 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#267 0x00007ffff632547d in pypy_g_CALL_METHOD__AccessDirect_star_1 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#268 0x00007ffff5a30c9d in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#269 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#270 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#271 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#272 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#273 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#274 0x00007ffff5bcf9c4 in pypy_g_exec_ ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#275 0x00007ffff575164a in pypy_g_BuiltinActivation_UwS_ObjSpace_W_Root_W_Root_W_R_2 () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#276 0x00007ffff59a16db in pypy_g_BuiltinCode_funcrun_obj ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#277 0x00007ffff5a2b2bf in pypy_g_CALL_FUNCTION__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#278 0x00007ffff5a30e57 in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#279 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#280 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#281 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#282 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#283 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#284 0x00007ffff632547d in pypy_g_CALL_METHOD__AccessDirect_star_1 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#285 0x00007ffff5a30c9d in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#286 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#287 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#288 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#289 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#290 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#291 0x00007ffff632547d in pypy_g_CALL_METHOD__AccessDirect_star_1 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#292 0x00007ffff5a30c9d in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#293 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#294 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#295 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#296 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#297 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#298 0x00007ffff632547d in pypy_g_CALL_METHOD__AccessDirect_star_1 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#299 0x00007ffff5a30c9d in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#300 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#301 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#302 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#303 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#304 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#305 0x00007ffff5a2b2bf in pypy_g_CALL_FUNCTION__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#306 0x00007ffff5a30e57 in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#307 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#308 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#309 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#310 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#311 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#312 0x00007ffff5bcf9c4 in pypy_g_exec_ ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#313 0x00007ffff575164a in pypy_g_BuiltinActivation_UwS_ObjSpace_W_Root_W_Root_W_R_2 () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#314 0x00007ffff59a16db in pypy_g_BuiltinCode_funcrun_obj ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#315 0x00007ffff5a2b2bf in pypy_g_CALL_FUNCTION__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#316 0x00007ffff5a30e57 in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#317 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#318 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#319 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#320 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#321 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#322 0x00007ffff5a2b2bf in pypy_g_CALL_FUNCTION__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#323 0x00007ffff5a30e57 in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#324 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#325 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#326 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#327 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#328 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#329 0x00007ffff5a2a53d in pypy_g_CALL_FUNCTION_EX__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#330 0x00007ffff5a3126d in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#331 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#332 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#333 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#334 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#335 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#336 0x00007ffff5a2b2bf in pypy_g_CALL_FUNCTION__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#337 0x00007ffff5a30e57 in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#338 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#339 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#340 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#341 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#342 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#343 0x00007ffff5a2a53d in pypy_g_CALL_FUNCTION_EX__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#344 0x00007ffff5a3126d in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#345 0x00007ffff59fd242 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#346 0x00007ffff60f60b0 in pypy_g_portal_11 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#347 0x00007ffff64d5ecb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter () from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#348 0x00007ffff59c484a in pypy_g_execute_frame ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#349 0x00007ffff68c5446 in __vmprof_eval_vmprof ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#350 0x00007ffff59a4a45 in pypy_g_call_function.star_3 ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#351 0x00007ffff5979959 in pypy_g_entry_point ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#352 0x00007ffff6a1c53f in pypy_main_startup ()
   from /opt/hostedtoolcache/PyPy/3.9.16/x64/bin/libpypy3.9-c.so
#353 0x00007ffff448ed90 in __libc_start_call_main (
    main=main@entry=0x401060 <main>, argc=argc@entry=7, 
    argv=argv@entry=0x7fffffffd7e8)
    at ../sysdeps/nptl/libc_start_call_main.h:58
#354 0x00007ffff448ee40 in __libc_start_main_impl (main=0x401060 <main>, 
    argc=7, argv=0x7fffffffd7e8, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fffffffd7d8)
    at ../csu/libc-start.c:392
#355 0x000000000040108e in _start ()
A debugging session is active.

	Inferior 1 [process 1952] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]

[The-Compiler]

Yeah, that looks like it crashes somewhere deep inside PyPy internals. I'm not really familiar with them, unfortunately. Perhaps it would make more sense to report this there?

[LilyFirefly]

I have reported this to the pypy project, with some further investigation of my own: https://foss.heptapod.net/pypy/pypy/-/issues/3960

[cfbolz]

thanks for investigating this, @LilyFoote! that gdb traceback is actually helpful to me! I'm going to try to find out what exactly is crashing.

it's likely something like: pytest assert rewriting produces an AST that is slightly different than what PyPy's ast-to-bytecode compiler expects. and instead of reporting that error sensibly, pypy just segfaults.

[cfbolz]

is there any chance we could try to pretty-print the AST in the CI run that segfaults, before it is passed into compile in the last line of _rewrite_test here?: https://github.com/pytest-dev/pytest/blob/main/src/_pytest/assertion/rewrite.py#L342

[cfbolz]

I trying to get this using a fork of pytest and @LilyFoote's reproducer repo, waiting for results here: https://github.com/cfbolz/pypy-segfault/actions/runs/5485589082/jobs/9994579372

[LilyFirefly]

@cfbolz You'll need to enable buildjet: https://buildjet.com/for-github-actions/docs/getting-started/run-your-first-workflow

Thankfully, they have a $5 credit so you don't need to pay until that runs out.

[cfbolz]

ah, I see. they still want a credit card, which I don't have. if I open a PR on your repo with my changes, we could run it on your repo, if that's ok for you @LilyFoote?

[cfbolz]

open one in any case: LilyFirefly/pypy-segfault#1