File descriptor error when subprocess call is used with event loop enabled in main thread

[ManojC]

BPO	38104
Nosy	@asvetlov, @1st1

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-09-13.11:57:12.834>
created_at = <Date 2019-09-11.10:12:22.124>
labels = ['type-crash', 'expert-asyncio']
title = 'File descriptor error when subprocess call is used with event loop enabled in main thread'
updated_at = <Date 2019-09-13.11:57:12.833>
user = 'https://bugs.python.org/ManojC'

bugs.python.org fields:

activity = <Date 2019-09-13.11:57:12.833>
actor = 'asvetlov'
assignee = 'none'
closed = True
closed_date = <Date 2019-09-13.11:57:12.834>
closer = 'asvetlov'
components = ['asyncio']
creation = <Date 2019-09-11.10:12:22.124>
creator = 'Manoj C'
dependencies = []
files = []
hgrepos = []
issue_num = 38104
keywords = []
message_count = 2.0
messages = ['351795', '352301']
nosy_count = 3.0
nosy_names = ['asvetlov', 'yselivanov', 'Manoj C']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue38104'
versions = ['Python 3.6']

[ManojC]

I have been using asyncio to run subprocess calls in a separate thread. For this purpose I start an event loop in my main thread as per the recommendation - https://docs.python.org/3/library/asyncio-subprocess.html#subprocess-and-threads .

Now when I use a normal subprocess call in the main thread, I start getting following file descriptor error after few iterations:

Exception ignored when trying to write to the signal wakeup fd:
BlockingIOError: [Errno 11] Resource temporarily unavailable

I have reproduced the problem in a small script and am seeing that the error goes away if I do not start the even loop in the main thread.

import asyncio
import subprocess
import time


def try_error():
    for i in range(1,500):
        print(i)
        try:
            subprocess.run(["ls", "-l"], check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        except subprocess.CalledProcessError as e:
            print(f"Exception raised {e.stderr}\nOutput {e.stdout}")



def definite_error():
    w = asyncio.get_child_watcher()
    l = asyncio.get_event_loop()
    try_error()


if __name__ == "__main__":
    definite_error()

This is the smallest subset of the code which can reproduce the error. In the original code, I run a asyncio.create_subprocess_exec in a parallel thread. The normal subprocess call is part of third party code which call from the main thread and hence cannot modify it.

[asvetlov]

You spawn too many subprocesses that finish virtually at the same time.
It leads to wakeup_fd overrun.

Python 3.6 is in security mode, sorry (and the fix is impossible).
Python 3.7 has warn_on_full_buffer=False flag for https://docs.python.org/3/library/signal.html#signal.set_wakeup_fd but asyncio doesn't use it for reasons.

Python 3.8 doesn't subscribe for SIGCHLD by default, thus the provided example finishes without any warning.

I'm going to closing the issue as won't fix.