Console breaks subprocesses stdout/stderr PIPE if script finishes before the subprocess

[Keou0007]

I have some code that launches a bunch of subprocesses and waits for return values. I'm trying to implement it in a way that the subprocesses will continue running and finish when I catch a KeyboardInterrupt.

I have code that achieves this, but I've noticed that the spyder console will kill the subprocesses if the launching python script exits before they finish. This only occurs in the spyder console and not in python or ipython run from the terminal. See below.

Original Code:

import time
import shlex
import subprocess as sp

start = time.time()
proc = []
ret = []
try:
    for i in range(3):
        proc.append(sp.Popen(shlex.split(f"sh test_sigint.sh {i}"), start_new_session=True)) # 
    
    print("All launched... hit CTRL-C now!")
    ret = [p.wait() for p in proc]
except KeyboardInterrupt:
    print("Caught SIGINT")
    ret = [p.wait() for p in proc]

print("rets:", ret, int(time.time()-start), 's')

the relevant bash script test_sigint.sh looks like:

#!/bin/bash
echo $1 starting shell script with PID $$
echo $1 sleeping for 10s
sleep 10
echo $1 finishing
exit $(( $1 * $1 ))

If I run this code and then hit CTRL-C while the subprocesses are sleeping, it waits for them to finish and gathers the return codes, then prints them and exits. This is what I expect.

I now edit the code so that it isn't waiting on the processes in the except block anymore

    ret = [p.wait() for p in proc]
except KeyboardInterrupt:
    print("Caught SIGINT")
    # ret = [p.wait() for p in proc]

What I expect to happen is that the python will print an empty list because the subprocesses haven't returned yet, but they will continue running in the background anyway. What happens, in the spyder console, is that as soon as the python script exits, the subprocesses are killed.

If I run exactly the same script in an ipython instance from the terminal, the script finishes how I expect, i.e. the subprocesses keep running and eventually exit.

So it seems something about spyder (or spyder-kernels i assume) is causing it to kill the subprocesses when it shouldn't.

I had wondered if maybe it was just preventing the subprocesses from printing to screen, so I played around with putting a touch statement at the end of the bash script and found that when run from spyder, those files never got touched.

Relevant details from conda list:

• spyder 4.0.0b2 py37_0 spyder-ide
• spyder-kernels 1.2.0 py37_0 spyder-ide
• ipython 7.5.0 py37h39e3cac_0
• python 3.7.3 h0371630_0

[ccordoba12]

Does this happen if you run your code in qtconsole instead of Spyder?

[Keou0007]

in qtconsole I get slightly different behaviour. The python print statements appear in the console, while the shell script outputs appear in the terminal I launched qtconsole from. Nonetheless the scripts finish like they do in ipython but don't in spyder.

[ccordoba12]

Ok, thanks for checking that out. We'll take a look at this as soon as we can.

[Keou0007]

I noticed something interesting that may or may not help. The documentation for subprocess says:

The arguments shown above are merely the most common ones, described below in Frequently Used Arguments (hence the use of keyword-only notation in the abbreviated signature). The full function signature is largely the same as that of the Popen constructor - most of the arguments to this function are passed through to that interface. (timeout, input, check, and capture_output are not.)

In my above code, is use Popen with start_new_session=True, which calls os.setsid() as a preexec function, preventing SIGINT from being passed to the subprocess. I've seen people say things like
subprocess.run(cmd).returncode is the equivalent of subprocess.Popen(cmd).wait(), so I figured I can probably give start_new_session to run and it will pass it on to Popen.

To test:

subprocess.run(cmd, start_new_session=True).returncode

subprocess dies when I hit CTRL-C.

subprocess.Popen(cmd, start_new_session=True).wait()

subprocess lives through KeyboardInterrupt and finishes.

I get the same result giving preexec_fn=os.setsid instead of start_new_session=True.
It seems to me that even though subprocess.run will happily accept all of the possible kwargs to Popen without giving a TypeError, it isn't actually passing all of them along on the backend. Whether this is by design or a bug I couldn't say. Is it likely this difference is what's effecting the spyder console here? or something similar?

[Keou0007]

I think I've nailed down the problem.
I've been launching the subprocess with threads now and through what seems to be some weirdness in how spyder handles daemonic threads, I was able to get the return codes of my subprocesses that I thought spyder was killing when the script exited.

def call_shell_script(num):
    print(f"start {current_thread().name}")
    ret = sp.Popen(shlex.split(f"sh test_sigint.sh {num}")).wait()
    print(f"end {current_thread().name} returned {ret}")
    return ret

threads = []
for i in range(3):
    t = Thread(target=call_shell_script, args=(i,))
    t.start()
    threads.append(t)

print('all started')

try:
    for t in threads:
        t.join()
except KeyboardInterrupt:
    print("SIGINT")

print("all done")

A few scenarios:

• Code as above: SIGINT is passed on to subprocesses, threads finish and print the return code for the subprocesses as -2. i.e. killed with SIGINT. That's what you'd expect.
• add start_new_session=True: SIGINT is not passed, script prints "all done" but doesn't exit because there are threads still running. shell script finishes and threads finish printing the correct return code.
• add t.daemon = True to make threads daemonic, run script from terminal: SIGINT is not passed to the sub processes, but when the python script gets to the end and all that's left is daemonic threads, it exits, killing the threads. The shell script finishes (printing to stdout), but the threads never print the return code because they've been killed.
• as above run in spyder: same thing happens but when the python script ends, the daemonic threads don't get killed (something to do with spyder-kernel?). As a result, the threads get to finish after the shell script exits, printing a return code of... -13 = SIGPIPE.

Turns out spyder wasn't killing my subprocesses, but was instead breaking the PIPE when the script exited. The shell script would then keep running until it reached it's next echo statement, then would find the broken pipe and exit with a SIGPIPE return code. The issue I had before is that I had no way of retrieving that return code, since the python script had already exited, so I just assumed it was killing the processes. The touch lines I was putting in to measure progress were after the echo lines, and so never got run.

I assume the bug here is something to do with how spyder is grabbing the stdout and/or subprocess.PIPE in order to print to the console.

[Keou0007]

A little unrelated, but while I've been playing with this, I also noticed that whatever spyder does to pipe stdout/stderr and print to the console isn't thread safe.
I often get multiple threads printing to the same line in the console, which doesn't happen if I run the same code anywhere else.

[ccordoba12]

I assume the bug here is something to do with how spyder is grabbing the stdout and/or subprocess.PIPE in order to print to the console.

Thanks for your thorough investigation of this problem. I think I know what's causing this now and I'll make a pull request to fix it soon.