With asyncio subprocess, send_signal() and the child process watcher will both call waitpid()

[lincheney]

BPO	43578
Nosy	@asvetlov, @1st1, @lincheney

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-03-21.06:53:15.574>
labels = ['type-bug', '3.9', 'expert-asyncio']
title = 'With asyncio subprocess, send_signal() and the child process watcher will both call waitpid()'
updated_at = <Date 2021-03-21.06:53:15.574>
user = 'https://github.com/lincheney'

bugs.python.org fields:

activity = <Date 2021-03-21.06:53:15.574>
actor = 'lincheney'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['asyncio']
creation = <Date 2021-03-21.06:53:15.574>
creator = 'lincheney'
dependencies = []
files = []
hgrepos = []
issue_num = 43578
keywords = []
message_count = 1.0
messages = ['389218']
nosy_count = 3.0
nosy_names = ['asvetlov', 'yselivanov', 'lincheney']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43578'
versions = ['Python 3.9']

Linked PRs

• gh-87744: fix waitpid race while calling send_signal in asyncio #121126
• [3.13] gh-87744: fix waitpid race while calling send_signal in asyncio (GH-121126) #121194

[lincheney]

Under unix, when creating a asyncio subprocess, the child process watcher will call waitpid() to reap the child, but if you call send_signal() (or terminate() or kill() ) on the asyncio subprocess, this will also call waitpid(), causing exactly one of these to fail, as you cannot call waitpid() on a PID more than once.

If the send_signal() fails, this doesn't seem much of an issue.
If the child process watcher fails however, it sets the returncode to 255 and also returns 255 when running wait() and also emits a warning.

I've seen this behaviour with the ThreadedChildWatcher, but possibly other Unix child watchers that use waitpid() suffer from the same problem.

The behaviour is racey (it depends on which one completes the waitpid() first), but if you try it enough it will appear:

import asyncio
import signal

async def main():
    while True:
        proc = await asyncio.create_subprocess_exec('sleep', '0.1')
        await asyncio.sleep(0.1)
        try:
            proc.send_signal(signal.SIGUSR1)
        except ProcessLookupError:
            pass
        assert (await proc.wait() != 255)

asyncio.run(main())

The output looks like:

Unknown child process pid 1394331, will report returncode 255
Traceback (most recent call last):
  File "/tmp/bob.py", line 14, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/tmp/bob.py", line 12, in main
    assert (await proc.wait() != 255)
AssertionError

This would be expected behaviour if I were explicitly calling waitpid() myself (ie I'm shooting my own foot, so I'd deserve the bad behaviour), but that's not the case here nor any other exotic code.

[FabianElsmer]

Did you find a workaround for this? I tried the different watcher classes asyncio provides using your example script, but all of them show this error after a while

[gvanrossum]

This one is new to me. I’m not sure how to approach it yet. The only workaround I can think of without digging in the code would be to ignore the error using try/except, but that’s a last resort.

[gvanrossum]

@graingert Any idea here?

[gvanrossum]

Oops, I see the problem isn't an exception so much as a lost exit status. I'm guessing we need to improve the communication between asyncio.subprocess.Process.wait() and the child watchers.

[kumaraditya303]

On my TODO list but low priority, I can review a PR fixing this though.

[kumaraditya303]

Child watcher will be removed in 3.14 at which point this will have to be reworked again so I'll fix this after child watchers are removed, it will be much simpler then.

[willingc]

Revisit after completion of #120804

[astrand]

This is a major issue for us, I am surprised that this issue is not getting more attention. It can be reproduced with the test case above on Python 3.9 and 3.10; please add corresponding labels. If I understand it correctly, the problem was introduced with https://bugs.python.org/issue38630 . Is it correct that the only workaround is to use os.kill instead of proc.terminate/proc.kill?

[gvanrossum]

This is a major issue for us, I am surprised that this issue is not getting more attention. It can be reproduced with the test case above on Python 3.9 and 3.10; please add corresponding labels. If I understand it correctly, the problem was introduced with https://bugs.python.org/issue38630 . Is it correct that the only workaround is to use os.kill instead of proc.terminate/proc.kill?

Temper your rhetoric, please. Just upgrade to Python 3.13.

[astrand]

This is a major issue for us, I am surprised that this issue is not getting more attention. It can be reproduced with the test case above on Python 3.9 and 3.10; please add corresponding labels. If I understand it correctly, the problem was introduced with https://bugs.python.org/issue38630 . Is it correct that the only workaround is to use os.kill instead of proc.terminate/proc.kill?

Temper your rhetoric, please. Just upgrade to Python 3.13.

Easier said than done. In our case the situation is the same as for most users, I believe: Python is provided by the platform. In our case, Yocto Kirkstone, which includes Python 3.10. In theory we could upgrade to the next Yocto LTS release which is Scarthgap, but it only includes Python 3.12; not 3.13.

The workaround with os.kill seems to work, though, at least with SafeChildWatcher.