Start the deprecation cycle for subprocess preexec_fn

[gpshead]

BPO	38435
Nosy	@gpshead, @vstinner, @diekhans, @markmentovai, @serhiy-storchaka, @izbyshev
PRs	gh-82616: Add process_group support to subprocess.Popen #23930 bpo-38435: Detect preexec_fn=os.setsid as start_new_session=True #23936

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = None
created_at = <Date 2019-10-10.18:18:17.382>
labels = ['type-feature', 'library', '3.11']
title = 'Start the deprecation cycle for subprocess preexec_fn'
updated_at = <Date 2022-03-16.16:54:37.730>
user = 'https://github.com/gpshead'

bugs.python.org fields:

activity = <Date 2022-03-16.16:54:37.730>
actor = 'markmentovai'
assignee = 'gregory.p.smith'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-10-10.18:18:17.382>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 38435
keywords = ['patch']
message_count = 18.0
messages = ['354397', '354404', '354407', '354439', '383708', '383720', '383724', '383725', '383726', '383727', '383728', '383736', '383740', '383863', '383988', '400498', '403026', '415352']
nosy_count = 6.0
nosy_names = ['gregory.p.smith', 'vstinner', 'diekhans', 'markmentovai', 'serhiy.storchaka', 'izbyshev']
pr_nums = ['23930', '23936']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue38435'
versions = ['Python 3.11']

[gpshead]

subprocess's preexec_fn feature needs to enter PendingDeprecationWarning state in 3.9, to become a DeprecationWarning in 3.10, with a goal of removing it in 3.11.

Rationale: We now live in a world full of threads, it is entirely unsafe to call back into the python interpreter within the forked child before exec per POSIX specification.

We've also already made preexec_fn no longer supported from CPython subinterpreters in 3.8.

If there are not already issues open for desired features of subprocess that do not yet have replacements or workarounds for *specific* actions that preexec_fn is being used for in your application, please open feature requests for those. (ex: calling umask is https://bugs.python.org/issue38417, and group, uid, gid setting has already landed in 3.9)

[vstinner]

What is the recommanded way to replace preexec_fn?

[gpshead]

With task specific arguments. cwd, start_new_session, group, extra_groups,
user, etc..

We cannot provide a general do everything replacement and should not try.
It not possible.

[vstinner]

We cannot provide a general do everything replacement and should not try. It not possible.

Well, I proposed a solution at:
https://bugs.python.org/issue38417#msg354242

I know that this solution has multiple flaws, but a bad solution may be better than no solution: breaking applications when upgrading to Python 3.11.

[diekhans]

calling setpgid() is a common post-fork task that would need to be an explicit parameter to Popen when preexec_fn goes away

[gpshead]

PR up to add setpgid support. From what I've come across, some setpgid() users can use setsid() already available via start_new_session= instead. But rather than getting into the differences between those, making both available makes sense to allow for anyone's case where setsid() isn't desired.

[gpshead]

https://bugs.python.org/issue42736 filed to track adding Linux prctl() support.

[gpshead]

Another preexec_fn use to consider:

 resource.setrlimit(resource.RLIMIT_CORE, (XXX, XXX))

Using an intermediate shell script wrapper that changes the rlimit and exec's the actual process is also an alternative.

[gpshead]

I'm also seeing a lot of os.setpgrp() calls, though those are more likely able to use start_new_session to do setsid() as a dropin replacement.

[gpshead]

signal.signal use case:

Calls to signal.signal(x, y) to sometimes to set a non SIG_DFL behavior on a signal. SIGINT -> SIG_IGN for example.

I see a lot of legacy looking code calling signal.signal in prexec_fn that appears to set SIG_DFL for the signals that Python otherwise modifies. Which restore_signals=True should already be doing.