Activity

1. graingert

   added

   type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

   on Nov 23, 2022
2. graingert commented on Nov 23, 2022

   

   graingert

   on Nov 23, 2022

   Last edited by graingert

   ContributorAuthor

   More actions

   see also bloomberg/memray#260 cc @pablogsal
3. graingert

   assigned

   pablogsal

   on Nov 23, 2022
4. pablogsal commented on Nov 23, 2022

   

   pablogsal

   on Nov 23, 2022

   Member

   More actions

   This looks like another frame lifetime issue. Here is the stack:

   Thread 1 "python" received signal SIGSEGV, Segmentation fault.
   [Switching to Thread 0x7ffff7cc5740 (LWP 223005)]
   0x00005555556cbc8b in PyFrame_GetCode (frame=<error reading variable: Cannot access memory at address 0x7ffff5e4c210>) at Objects/frameobject.c:1303
   1303    Objects/frameobject.c: Directory not empty.
   (gdb) up
   #1  frame_getcode (f=<error reading variable: Cannot access memory at address 0x7ffff5e4c210>, closure=<optimized out>)
       at Objects/frameobject.c:97
   97      in Objects/frameobject.c
   (gdb)
   #2  0x0000555555702309 in _PyObject_GenericGetAttrWithDict (Python Exception <class 'gdb.error'>: There is no member named f_frame.
   obj=, name='f_code', dict=0x0, suppress=0) at ./Include/object.h:133
   133     ./Include/object.h: Directory not empty.
   (gdb)
   #3  0x0000555555701836 in PyObject_GetAttr (Python Exception <class 'gdb.error'>: There is no member named f_frame.
   v=v@entry=, name='f_code') at Objects/object.c:916
   916     Objects/object.c: Directory not empty.
   (gdb)
   #4  0x00005555556440a1 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>)
       at Python/ceval.c:3466
   3466    Python/ceval.c: Directory not empty.
   (gdb)
   #5  0x00005555556c1bd5 in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff4840330, tstate=0x555555adfa18 <_PyRuntime+166328>)
       at ./Include/internal/pycore_ceval.h:73
   73      ./Include/internal/pycore_ceval.h: Directory not empty.
   (gdb)
   #6  gen_send_ex2 (closing=0, exc=0, presult=<synthetic pointer>, arg=0x0, gen=0x7ffff48402e0) at Objects/genobject.c:219
   219     Objects/genobject.c: Directory not empty.
   (gdb)
   #7  gen_iternext (gen=0x7ffff48402e0) at Objects/genobject.c:594
   
5. pablogsal commented on Nov 23, 2022

   

   pablogsal

   on Nov 23, 2022

   Member

   More actions

   CC: @markshannon @brandtbucher
6. pablogsal commented on Nov 23, 2022

   

   pablogsal

   on Nov 23, 2022

   Member

   More actions

   I can confirm that still happens with 3.11 current.
7. brandtbucher commented on Nov 23, 2022

   

   brandtbucher

   on Nov 23, 2022

   Member

   More actions

   Thanks, I'll look at this today (Mark's out right now).
8. brandtbucher

   self-assigned this

   on Nov 23, 2022
9. graingert commented on Nov 23, 2022

   

   graingert

   on Nov 23, 2022

   ContributorAuthor

   More actions

   I've now eliminated pytest from my reproducer: graingert/segfault-repro@34b5962

   ❤️React with ❤️4Reacted by Pradyun Gedam, Brandt Bucher, Thomas Grainger and Matt Wozniski
10. brandtbucher commented on Nov 23, 2022

    

    brandtbucher

    on Nov 23, 2022

    Member

    More actions

    I've now eliminated pytest from my reproducer: graingert/segfault-repro@34b5962

    Oh thank God. You have no idea how helpful this is.
11. brandtbucher commented on Nov 23, 2022

    

    brandtbucher

    on Nov 23, 2022

    Last edited by brandtbucher

    Member

    More actions

    @graingert, how should I run the new reproducer? I'm unable to get 3.11 head to crash with pip install -r requirements.txt; python test_ssltransport.py.
12. brandtbucher commented on Nov 23, 2022

    

    brandtbucher

    on Nov 23, 2022

    Member

    More actions

    I see the segfault intermittently when running pytest, though.
13. brandtbucher commented on Nov 23, 2022

    

    brandtbucher

    on Nov 23, 2022

    Member

    More actions

    In the meantime, I'm using pytest to reproduce. I've gotten test_ssltransport.py down to 270 lines, removed __init__.py entirely, and gotten pyproject.toml down to just:

    [tool.pytest.ini_options]
    filterwarnings = ["error"]
14. brandtbucher commented on Nov 23, 2022

    

    brandtbucher

    on Nov 23, 2022

    Member

    More actions

    Let me know if you want me to open a PR with the simplified reproducers (or, even better, if you want me to just push them).
15. graingert commented on Nov 23, 2022

    

    graingert

    on Nov 23, 2022

    ContributorAuthor

    More actions

    @graingert, how should I run the new reproducer? I'm unable to get 3.11 head to crash with pip install -r requirements.txt; python test_ssltransport.py.

    You need to run it with python -W error
16. 18 remaining items

    Load more

17. pablogsal commented on Nov 30, 2022

    

    pablogsal

    on Nov 30, 2022

    Last edited by pablogsal

    Member

    More actions

    I still don't undetstand why this doesn't crash:

    import sys
    
    class DelRaises:
        def __del__(self):
            global sneaky
            sneaky = sys._getframe()
    
    def run():
        _ = DelRaises()
    
    print("Running...")
    run()
    print("Crashing...")
    

    Unfortunately, I am currently dealing with a health issue so I could not find time to deal with this in detail :(
18. brandtbucher commented on Nov 30, 2022

    

    brandtbucher

    on Nov 30, 2022

    Member

    More actions

    Sorry, to hear that. :(

    I was able to figure this out yesterday, but was waiting to share in the meeting. I’ll post here in case you can’t come:

    I was able to get a single-threaded version to crash, too. It’s just easier to do with multiple threads since the frame object has a stale pointer into the thread’s frame stack. In our threaded example, after the thread finishes, the frame stack ceases to exist, so things can go south pretty quickly. When single-threaded, we just have a pointer into an old part of the current chunk, so things appear valid for longer if you don’t try to do much.

    I’m on my phone now, but if I remember correctly, doing something like sneaky.f_back.f_locals after the run() makes the single-threaded version crash.

    Basically, we should mark this frame as “incomplete” (or just unlink it, not sure yet which is easier) after checking for existing frame objects, but before clearing stuff. This restores the behavior of 3.10, which made it look like __del__ was called from run’s caller, not from run itself.

    I’ll have a PR up for review this week.

    👍React with 👍3Reacted by Pablo Galindo Salgado, Thomas Grainger and Łukasz Langa
19. pablogsal

    added

    release-blocker

    on Dec 3, 2022
20. ezio-melotti

    added this to Release and Deferred blockers 🚫on Dec 3, 2022
21. ezio-melotti

    moved this to Todo in Release and Deferred blockers 🚫on Dec 3, 2022
22. pablogsal commented on Dec 3, 2022

    

    pablogsal

    on Dec 3, 2022

    Member

    More actions

    Marking this as 3.11 release blocker

    👀React with 👀1Reacted by Łukasz Langa
23. brandtbucher

    added

    3.11only security fixesonly security fixes

    3.12only security fixesonly security fixes

    on Dec 5, 2022
24. pablogsal

    added 2 commits that reference this issue on Dec 6, 2022

    GH-99729: Unlink frames before clearing them (GH-100030)

    b72014c

    [3.11] GH-99729: Unlink frames before clearing them (#100047)

    2182a71
25. pablogsal commented on Dec 6, 2022

    

    pablogsal

    on Dec 6, 2022

    Last edited by pablogsal

    Member

    More actions

    PRs merged so I am closing the issue. Thanks a lot to everyone that participated in this bug, from identifying it to fixing it. You all rock 🤘
26. pablogsal

    removed

    release-blocker

    on Dec 6, 2022
27. pablogsal

    closed this as completedon Dec 6, 2022
28. Repository owner moved this from Todo to Done in Release and Deferred blockers 🚫on Dec 6, 2022
29. brandtbucher commented on Dec 6, 2022

    

    brandtbucher

    on Dec 6, 2022

    Member

    More actions

    I've confirmed that the 3.11 patch fixes the full original reproducer, as well.