Skip to content

Ctrl-C interrupts not working inside python execution #527

Open
Open
Ctrl-C interrupts not working inside python execution#527
Labels
bugSomething isn't working

Description

@axsk
axsk
opened on Jul 19, 2024

Affects: PythonCall / JuliaCall

Describe the bug
Interrupting during python execution is not working properly (not even on one thread).
Keeping ctrl+c pressed interrupts the execution, but the next python call segfaults julia.
This applies to both, PythonCall and PyCall.

julia> using PyCall                                                                                                                                                                                          
                                                                                                                                                                                                             
julia> py"""                                                                                                                                                                                                 
       import time                                                                                                                                                                                           
       for i in range(100):                                                                                                                                                                                  
           time.sleep(1)"""                                                                                                                                                                                  
^C^C^C^C^C^C^C^C^C^C^CWARNING: Force throwing a SIGINT                                                                                                                                                       
^C^CERROR: InterruptException:                                                                                                                                                                               
Stacktrace:                                                                                                                                                                                                  
 [1] macro expansion                                                                                                                                                                                         
   @ ~/.julia/packages/PyCall/1gn3u/src/exception.jl:108 [inlined]                                                                                                                                           
 [2] #117                                                                                                                                                                                                    
   @ ~/.julia/packages/PyCall/1gn3u/src/pyeval.jl:38 [inlined]                                                                                                                                               
 [3] disable_sigint                                                                                                                                                                                          
   @ ./c.jl:473 [inlined]                                                                                                                                                                                    
 [4] pyeval_(s::String, globals::PyDict{String, PyObject, true}, locals::PyDict{String, PyObject, true}, input_type::Int64, fname::String)                                                                   
   @ PyCall ~/.julia/packages/PyCall/1gn3u/src/pyeval.jl:37                                                                                                                                                  
 [5] macro expansion                                                                                                                                                                                         
   @ ~/.julia/packages/PyCall/1gn3u/src/pyeval.jl:230 [inlined]                                                                                                                                              
 [6] top-level scope                                                                                                                                                                                         
   @ REPL[2]:1

julia> py"""
       import time
       for i in range(100):
           time.sleep(1)"""

[473012] signal (11.1): Segmentation fault
in expression starting at REPL[2]:1
_PyInterpreterState_GET at /usr/local/src/conda/python-3.10.14/Include/internal/pycore_pystate.h:117 [inlined]
PyUnicode_DecodeFSDefaultAndSize at /usr/local/src/conda/python-3.10.14/Objects/unicodeobject.c:4095
Py_CompileStringExFlags at /usr/local/src/conda/python-3.10.14/Python/pythonrun.c:1390
macro expansion at /home/htc/bzfsikor/.julia/packages/PyCall/1gn3u/src/exception.jl:108 [inlined]
pyeval_ at /home/htc/bzfsikor/.julia/packages/PyCall/1gn3u/src/pyeval.jl:34

julia> using PythonCall                                                                                                                                                                                                                                                                                                                                                                       
                                                                                                                                                                                                             
julia> @pyexec """                                                                                                                                                                                           
              import time                                                                                                                                                                
              for i in range(100):                                                                                                                                                           
                time.sleep(1)"""                                                                                                                                                                             
                                                                                                                                                                                                             
^C^C^C^C^CWARNING: Force throwing a SIGINT                                                                                                                                                                   
^CERROR: InterruptException:                                                                                                                                                                                 
Stacktrace:                                                                                                                                                                                                  
 [1] PyObject_CallObject                                                                                                                                                                                     
   @ ~/.julia/packages/PythonCall/S5MOg/src/C/pointers.jl:297 [inlined]                                                                                                                                      
 [2] macro expansion                                                                                                                                                                                         
   @ ~/.julia/packages/PythonCall/S5MOg/src/Core/Py.jl:132 [inlined]                                                                                                                                         
 [3] pycallargs(f::Py, args::Py)                                                                                                                                                                             
   @ PythonCall.Core ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:212                                                                                                                             
 [4] pycall(::Py, ::Py, ::Vararg{Py}; kwargs::@Kwargs{})                                                                                                                                                     
   @ PythonCall.Core ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:230                                                                                                                             
 [5] pycall                                                                                                                                                                                                  
   @ ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:220 [inlined]                                                                                                                                   
 [6] Py                                                                                                                                                                                                      
   @ ~/.julia/packages/PythonCall/S5MOg/src/Core/Py.jl:339 [inlined]                                                                                                                                         
 [7] pyexec(::Type{Nothing}, code::Py, globals::Module, locals::Tuple{})                                                                                                                                     
   @ PythonCall.Core ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:1212                                                                                                                            
 [8] top-level scope                                                                                                                                                                                         
   @ ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:1260                                                                                                                                            
                                                                                                                                                                                                             
julia> @pyexec """                                                                                                                                                                                           
              import time                                                                                                                                                                                           
              for i in range(100):                                                                                                                                                                   
                time.sleep(1)"""                                                                                                                                                                             
                                                                                                                                                                                                             
[472692] signal (11.1): Segmentation fault                                                                                                                                                                   
in expression starting at REPL[2]:1                                                                                                                                                                          
_PyInterpreterState_GET at /usr/local/src/conda/python-3.12.4/Include/internal/pycore_pystate.h:133 [inlined]                                                                                                
maybe_freelist_pop at /usr/local/src/conda/python-3.12.4/Objects/tupleobject.c:1133 [inlined]                                                                                                                
tuple_alloc at /usr/local/src/conda/python-3.12.4/Objects/tupleobject.c:43 [inlined]                                                                                                                         
PyTuple_New at /usr/local/src/conda/python-3.12.4/Objects/tupleobject.c:74                                                                                                                                   
PyTuple_New at /home/htc/bzfsikor/.julia/packages/PythonCall/S5MOg/src/C/pointers.jl:297 [inlined]                                                                                                           
pynulltuple at /home/htc/bzfsikor/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:824 [inlined]

Your system
Please provide detailed information about your system:

  • Linux, Julia 1.10.4, both PyCall and PythonCall on newest versions.

Activity

  1. axsk
    added
    bugSomething isn't working
    on Jul 19, 2024

  2. cjdoris commented on Jul 19, 2024

    @cjdoris
    cjdoris
    on Jul 19, 2024
    Member

    It's true that Python code is not currently interruptible when called from PythonCall. That requires some signal handling magic that I think isn't currently possible in Python and Julia.

    In your example you threw SIGINT by holding down Ctrl-C, I'm not surprised this would crash things as it's generally intended as a last resort to stop a program.

    What happens if you just press Ctrl-C once and wait?

  3. kleinschmidt commented on Aug 15, 2024

    @kleinschmidt
    kleinschmidt
    on Aug 15, 2024

    I can reliably get a segfault on one Ctrl-C, which looks like this:

    [ Info: Received message: nothing
^C
[82624] signal (2): Interrupt: 2
in expression starting at /Users/dkleinschmidt/work/REDACTED/demo.jl:22
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
Allocations: 3128814 (Pool: 3125853; Big: 2961); GC: 4

[82624] signal (11.2): Segmentation fault: 11
in expression starting at /Users/dkleinschmidt/work/REDACTED/demo.jl:22
_PyUnicode_FromId at /Users/dkleinschmidt/.pyenv/versions/3.10.13/lib/libpython3.10.dylib (unknown line)
_PyImport_GetModuleId at /Users/dkleinschmidt/.pyenv/versions/3.10.13/lib/libpython3.10.dylib (unknown line)
wait_for_thread_shutdown at /Users/dkleinschmidt/.pyenv/versions/3.10.13/lib/libpython3.10.dylib (unknown line)
Py_FinalizeEx at /Users/dkleinschmidt/.pyenv/versions/3.10.13/lib/libpython3.10.dylib (unknown line)
Py_FinalizeEx at /Users/dkleinschmidt/.julia/packages/PythonCall/flx5V/src/C/pointers.jl:303 [inlined]
#37 at /Users/dkleinschmidt/.julia/packages/PythonCall/flx5V/src/C/context.jl:194
unknown function (ip: 0x139e3444f)
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:3077
_atexit at ./initdefs.jl:428
jfptr__atexit_79411.3 at /Users/dkleinschmidt/.julia/juliaup/julia-1.10.4+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/./julia.h:1982 [inlined]
ijl_atexit_hook at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/init.c:280
jl_exit_thread0_cb at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/./signals-mach.c:464
Allocations: 3128814 (Pool: 3125853; Big: 2961); GC: 4

    I can't share the exact code unfortunately, but there's no multi-threading happening in julia and the python code that is executing is a call to confluent_python.Consumer.poll() (which ultimately is a shim for a librdkafka C function call...). I do think that librdkafka is multithreaded though if that makes a difference...

    This is with

    • Julia 1.9.4 1.10.4
    • PythonCall 0.9.22
    • confluent_kafka 2.5.0
    • Python 3.10.13
    • MacOS 14.5 (23F79) (Macbook Air M1)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions