test_frame does crash randomly on Linux (x86-64, ppc64le, s390x)

[vstinner]

Crash report

Example: https://buildbot.python.org/#/builders/64/builds/9301

test_repr_deep (test.test_frame.FrameLocalsProxyMappingTests.test_repr_deep) ...

Fatal Python error: Segmentation fault

Current thread 0x00007f2ed21d5400 [python] (most recent call first):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/unittest/case.py", line 247 in handle
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/unittest/case.py", line 813 in assertRaises
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/test/mapping_tests.py", line 634 in test_repr_deep
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/unittest/case.py", line 615 in _callTestMethod
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/unittest/case.py", line 669 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/unittest/case.py", line 725 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/unittest/suite.py", line 122 in run
  ...

Linked PRs

• gh-133261: Use __builtin_frame_address() on GCC 9 and older #133269
• gh-133261: Increase C stack margin #133401
• GH-133261: Make sure that the GC doesn't untrack objects in trashcan #133431
• GH-133261: Make sure trashcan pointers look mortal -- 32 bit #133450

[vstinner]

The crash might be related to the recent trashcan change: 44e4c47

[vstinner]

Extract of the gdb backtrace:

#0  _PyObject_GC_UNTRACK (op=0x7fffdbc073d0) at ./Include/internal/pycore_gc.h:279
#1  listiter_dealloc (self=0x7fffdbc073d0) at Objects/listobject.c:3997
#2  0x00000000004f1a2e in _PyTrash_thread_destroy_chain (tstate=<optimized out>) at Objects/object.c:2957
#3  _Py_Dealloc (op=op@entry=0x7fffdbbf7f40) at Objects/object.c:3073
#4  0x00000000004aeda8 in Py_DECREF (op=0x7fffdbbf7f40) at ./Include/refcount.h:433
#5  framelocalsproxy_repr (self=0x7fffdefaf760) at Objects/frameobject.c:532
#6  framelocalsproxy_repr (self=0x7fffdefaf760) at Objects/frameobject.c:512
#7  0x00000000004f1f4a in PyObject_Repr (v=0x7fffdefaf760) at ./Include/object.h:270
...
#310188 0x00000000004f1f4a in PyObject_Repr (v=0x7fffdc742b40) at ./Include/object.h:270
#310189 PyObject_Repr (v=v@entry=0x7fffdc742b40) at Objects/object.c:753
#310190 0x00000000004aed74 in framelocalsproxy_repr (self=0x7fffdc72cf10) at Objects/frameobject.c:531
#310191 framelocalsproxy_repr (self=0x7fffdc72cf10) at Objects/frameobject.c:512
#310192 0x00000000004f1f4a in PyObject_Repr (v=0x7fffdc72cf10) at ./Include/object.h:270
#310193 PyObject_Repr (v=0x7fffdc72cf10) at Objects/object.c:753
#310194 0x0000000000564e2b in PyUnicodeWriter_WriteRepr (writer=writer@entry=0x7fffdc742af0, obj=<optimized out>) at Objects/unicodeobject.c:13969
#310195 0x00000000004d7bd7 in dict_repr_lock_held (self=0x7fffdc742ac0) at Objects/dictobject.c:3372
#310196 dict_repr (self=0x7fffdc742ac0) at Objects/dictobject.c:3401
#310197 0x00000000004f1f4a in PyObject_Repr (v=0x7fffdc742ac0) at ./Include/object.h:270
#310198 PyObject_Repr (v=v@entry=0x7fffdc742ac0) at Objects/object.c:753
#310199 0x00000000004aed74 in framelocalsproxy_repr (self=0x7fffdc72cf40) at Objects/frameobject.c:531
#310200 framelocalsproxy_repr (self=0x7fffdc72cf40) at Objects/frameobject.c:512
#310201 0x00000000004f1f4a in PyObject_Repr (v=0x7fffdc72cf40) at ./Include/object.h:270
...
#310486 0x000000000067bb72 in Py_RunMain () at Modules/main.c:767

[ambv]

Also happened randomly on my Windows 11 buildbot:
https://buildbot.python.org/#/builders/1088/builds/722/steps/5/logs/stdio

[vstinner]

Also happened randomly on my Windows 11 buildbot:
https://buildbot.python.org/#/builders/1088/builds/722/steps/5/logs/stdio

Oh strange. This bug looks more like #133199 : assertion failure in _PyObject_GC_UNTRACK().

test_repr_deep (test.test_frame.FrameLocalsProxyMappingTests.test_repr_deep) ...

R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Objects\listobject.c:3997: _PyObject_GC_UNTRACK: Assertion "_PyObject_GC_IS_TRACKED(((PyObject*)(op)))" failed: object not tracked by the garbage collector
Enable tracemalloc to get the memory block allocation traceback
object address  : 0000016FA81B1090
object refcount : 0
object type     : 00007FFB97E7EEB0
object type name: list_iterator
object repr     : <refcnt 0 at 0000016FA81B1090>
Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized
Warning -- Unraisable exception
Exception ignored in the internal traceback machinery:
Traceback (most recent call last):
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\traceback.py", line 143, in _print_exception_bltin
    return print_exception(exc, limit=BUILTIN_EXCEPTION_LIMIT, file=file, colorize=colorize)
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\traceback.py", line 134, in print_exception
    te.print(file=file, chain=chain, colorize=colorize)
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\traceback.py", line 1577, in print
    for line in self.format(chain=chain, colorize=colorize):
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\traceback.py", line 1514, in format
    yield from _ctx.emit(exc.format_exception_only(colorize=colorize))
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\traceback.py", line 1001, in emit
    for text in text_gen:
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\traceback.py", line 1268, in format_exception_only
    isinstance(self.__notes__, collections.abc.Sequence)
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\abc.py", line 119, in __instancecheck__
    return _abc_instancecheck(cls, instance)
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\abc.py", line 123, in __subclasscheck__
    return _abc_subclasscheck(cls, subclass)
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\abc.py", line 123, in __subclasscheck__
    return _abc_subclasscheck(cls, subclass)
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\abc.py", line 123, in __subclasscheck__
    return _abc_subclasscheck(cls, subclass)
RecursionError: Stack overflow (used 11719 kB)
Current thread 0x00000408 (most recent call first):
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\case.py", line 247 in handle
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\case.py", line 813 in assertRaises
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\mapping_tests.py", line 634 in test_repr_deep
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\case.py", line 615 in _callTestMethod
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\case.py", line 669 in run
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\case.py", line 725 in __call__
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\suite.py", line 122 in run
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\suite.py", line 84 in __call__
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\suite.py", line 122 in run
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\suite.py", line 84 in __call__
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\unittest\runner.py", line 259 in run
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\single.py", line 84 in _run_suite
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\single.py", line 42 in run_unittest
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\single.py", line 162 in test_func
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\single.py", line 118 in regrtest_runner
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\single.py", line 165 in _load_run_test
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\single.py", line 210 in _runtest_env_changed_exc
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\single.py", line 319 in _runtest
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\single.py", line 348 in run_single_test
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\worker.py", line 92 in worker_process
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\worker.py", line 127 in main
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\test\libregrtest\worker.py", line 131 in <module>
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\runpy.py", line 88 in _run_code
  File "R:\buildarea\pull_request.ambv-bb-win11.bigmem\build\Lib\runpy.py", line 198 in _run_module_as_main
Extension modules: _testinternalcapi, _testcapi (total: 2)

[vstinner]

@ambv: Would you mind to open a separated issue for the Windows failure? It's a different root cause related to _PyObject_GC_UNTRACK().

[encukou]

Since the PR was merged, the s390x RHEL8 buildbot started failing: https://buildbot.python.org/#/builders/442/builds/9249

[StanFromIreland]

Can also reproduce (~25% of the time) on my machine, x86-64 Fedora 42.

[vstinner]

The crash or the GC untrack assertion error?