Repository navigation
potential undefined behavior with subprocess using vfork() on Linux? #91401
Description
| BPO | 47245 |
|---|---|
| Nosy | @gpshead, @socketpair, @izbyshev |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/gpshead'
closed_at = <Date 2022-04-08.13:33:50.215>
created_at = <Date 2022-04-06.20:24:52.654>
labels = ['extension-modules', 'type-bug', '3.10', '3.11']
title = 'potential undefined behavior with subprocess using vfork() on Linux?'
updated_at = <Date 2022-04-08.13:33:50.214>
user = 'https://github.com/socketpair'bugs.python.org fields:
activity = <Date 2022-04-08.13:33:50.214>
actor = 'socketpair'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2022-04-08.13:33:50.215>
closer = 'socketpair'
components = ['Extension Modules']
creation = <Date 2022-04-06.20:24:52.654>
creator = 'socketpair'
dependencies = []
files = []
hgrepos = []
issue_num = 47245
keywords = ['3.10regression']
message_count = 13.0
messages = ['416897', '416898', '416899', '416900', '416904', '416905', '416909', '416917', '416946', '416964', '416974', '416976', '416984']
nosy_count = 3.0
nosy_names = ['gregory.p.smith', 'socketpair', 'izbyshev']
pr_nums = []
priority = 'normal'
resolution = 'works for me'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue47245'
versions = ['Python 3.10', 'Python 3.11']
Activity
socketpair commentedon Apr 6, 2022
Using vfork in bpo-35823 is VERY tricky. Please comment out vfork() usage for now. Yes, we can (should) use vfork(), but we have to rewrite the child code.
https://bugzilla.kernel.org/show_bug.cgi?id=215813
I would say it's URGENT.
socketpair commentedon Apr 6, 2022
Rewriting in a way that guarantee no stack (and heap) usage. Because stack is shared between child and parent. It seems there is no crossplatform way. Happily we can use some code like I wrote by link in the first message. OR, stick to posix_nspawn which is cross-platform.
socketpair commentedon Apr 6, 2022
cpython/Modules/_posixsubprocess.c
Line 717 in 4a08c4c
----------------
Py_NO_INLINE static void
child_exec(char *const exec_array[],
char *const argv[],
char *const envp[],
const char *cwd,
int p2cread, int p2cwrite,
int c2pread, int c2pwrite,
-------------
So, calling child_exec pushes to stack GUARANTEED. This is the bug. In fact everything works, but is too weak and break at any moment.
Again. Please comment calling vfork() for now. And re-implement child part later.
socketpair commentedon Apr 6, 2022
Solution:
https://github.com/bminor/glibc/blob/master/sysdeps/unix/sysv/linux/spawni.c#L309
In short - do not use vfork(). Use clone(CLONE_VM | CLONE_VFORK). and do something with stack.
gpshead commentedon Apr 6, 2022
Can you provide a reproducable way to demonstrate evidence of a problem in CPython's use of the Linux libc vfork() implementation? A test case that causes a CPython parent or child process on Linux when built with HAVE_VFORK failing to function properly would help prioritize this because in practice nobody has reported problems in 3.10.
(we've deployed the subprocess vfork code into thousands production Python programs at work in the past year w/o issues being reported - though we have a constrained environment with use on only a couple of libc versions and limited set of kernels on a few very common architectures)
General thinking (possible dated and incorrect - against what https://man7.org/linux/man-pages/man2/vfork.2.html wording claims with its "or calls any other function" text):
Pushing additional data onto the stack in the child process _should_ not a problem. That by definition lands in previously unused pre-allocated stack space. If that page faults, that could map a new page into the process state shared by both the paused parent and running child. But this page mapping should be fine - the child exec that resumes the parent means the parent is the only one who sees it.
When the parent process resumes, sure, that data will be in that memory on the unallocated portion of stack, but critically the *stack pointer* in the parent process (a register) never changes. As far as I understand things, registers are not shared between vfork()ed processes. So the parent only sees some temporary data having been written to the unused region of the stack by the since-replaced by exec() child process. No big deal.
**Untrue wishful thinking**: If a new stack were needed on a given platform for use in the vfork()ed child, I'd like it to be the job of libc to take care of that for us. glibc sources do no such thing (every vfork supporting architecture has a vfork.S code that appears to make the syscall and jump back to the caller without stack manipulation).
[-]Subprocess with vfork() is broken[/-][+]potential undefined behavior with subprocess using vfork() on Linux[/+]gpshead commentedon Apr 6, 2022
Immediate action item: Add a way for people to disable vfork at runtime by setting a flag in the subprocess module, just in case.
This can be backported to 3.10 - It'd provide an escape hatch for anyone without a need to rebuild Python to disable use of vfork() without resorting to LD_PRELOAD hacks.
This is not an urgent issue unless actual practical problems are being observed in the field.
gpshead commentedon Apr 6, 2022
Our current assumptions around the use of vfork() are very much glibc specific.
Another useful reference for reasoning, comments, and history is https://github.com/golang/go/blob/master/src/syscall/exec_linux.go#L146
forkAndExecInChild1izbyshev commentedon Apr 7, 2022
In short: both this bug report and [1] are invalid.
The reason why doing syscall(SYS_vfork) is illegal is explained by Florian Weimer in [2]:
This is off-topic here because CPython calls vfork() via libc, but I'll still expand the Florian's comment. Suppose one wants to write my_vfork() wrapper over vfork syscall. When vfork syscall returns the first time, my_vfork() has to return to its caller. This necessarily involves knowing the return address. On some architectures this return address is stored on the stack by the caller (e.g. x86). The problem is that any data in my_vfork() stack frame can be overwritten by its caller once it returns the first time. Then, when vfork syscall returns the second time, my_vfork() could be unable to return to its caller because the data it fetches from its (now invalid) stack frame is garbage. This is precisely what happens when one implements my_vfork() as syscall(SYS_vfork). To avoid this, the most common strategy is to store the return address into a register that's guaranteed to be preserved around syscall by the OS ABI. For example, the x86-64 musl implementation [3] stores the return address in rdx (which is preserved around syscall) and then restores it after syscall (both on the first and the second return of the syscall).
Now back to CPython. The real problem with stack sharing between the child and the parent could be due to compiler bugs, e.g. if a variable stored on the stack is modified in the child branch of "if (vfork())", but the compiler assumes that some other variable sharing the stack slot with the first one is *not* modified in the parent branch (i.e. after vfork() returns the second time). But all production compilers handle vfork() (and setjmp(), which has a similar issue) in a special way to avoid this, and GCC has __attribute__((returns_twice)) that a programmer could use for custom functions behaving in this way (my_vfork() would have to use this attribute).
Regarding a separate stack for the child and clone(CLONE_VM|CLONE_VFORK), I considered this in bpo-35823, but this has its own problems. The most important one is that CPython would be in business of choosing the correct stack size for the child's stack, but CPython is *not* in control of all the code executing in the child because it calls into libc. In practice, people use various LD_PRELOAD-based software that wraps various libc functions (e.g. Scratchbox 2 build environment for Sailfish OS is an LD_PRELOAD-based sandbox), so even common-sense assumptions like "execve() in libc can't use a lot of stack!" might turn out to be wrong. CPython *could* work around that by using direct syscalls for everything in the child, or by using some "large" size that "should be enough for everybody", but I wouldn't want to see that unless we have a real problem with vfork(). Note that vfork()-based posix_spawn() implementations in C libraries do *not* have this problem because they fully control all code in the child (e.g. they would use a non-interposable execve() symbol or a direct syscall).
I don't think any action is needed at all, and I think this issue should be closed.
Could you clarify what glibc-specific assumptions you mean? In bpo-35823 I tried to use as little assumptions as possible.
[1] https://bugzilla.kernel.org/show_bug.cgi?id=215813
[2] https://bugzilla.kernel.org/show_bug.cgi?id=215813#c2
[3] https://git.musl-libc.org/cgit/musl/tree/src/process/x86_64/vfork.s?id=ced75472d7e3d73d5b057e36ccbc7b7fcba95104
gpshead commentedon Apr 7, 2022
Thanks! I agree with you that this is probably not an actual problem on Linux.
_I did look at the various glibc architecture vfork.s implementations: Cute tricks used on some where they need to avoid a stack modifying traditional return from vfork()._
As for glibc specifics, I'm mostly thinking of the calls we do in the child.
According to the "Standard Description (POSIX.1)" calls to anything other than
_exit()orexec*()are not allowed. But the longer "Linux Description" in that vfork(2) man page does not say that. Which implies merely by omission that calls to other things are okay so long as you understand everything they do to the process heap/stack/state. (I wish it were explicit about that)Some of the calls we do from our child_exec() code... many are likely "just" syscall shims and thus fine - but that is technically up to libc.
A few others are Py functions that go elsewhere in CPython and while they may be fine for practical reasons today with dangerous bits on conditional branches that technically should not be possible to hit given the state by the time we're at this point in _posixsubprocess, pose a future risk - anyone touching implementations of those is likely unaware of vfork'ed child limitations that must be met.
For example if one of the potential code paths that trigger an indirect Py_FatalError() is hit... that fatal exit code is definitely not post-vfork-child safe. The pre-exec child dying via that could screw up the vfork parent process's state.
8 remaining items
gpshead commentedon Apr 13, 2022
I'm reopening this to track adding a failsafe so people have a big-red-button style way to disable Python's use of vfork without recompiling just in case of future problems. That arguably should've been there from the start in 3.10. Though we still have no known need to use it, by the time you do it needs to already exist.
I'll do a manual more conservative backport of the disable_vfork_reason button to 3.10 for 3.10.5.
vstinner commentedon Apr 19, 2022
I wish posix_spawn() could be used in more cases on Linux: see #86904.
gh-91401: Add a failsafe way to disable vfork. (#91490)
izbyshev commentedon May 1, 2022
On the contrary, I wish posix_spawn() completely disabled on Linux. Using different backends depending on Popen() arguments only adds possibility for hard-to-understand behavioral differences. And posix_spawn() in glibc is still buggy (#91307 (comment)).
MojoVampire commentedon May 1, 2022
@gpshead: It looks like you added the big-red-button; can this be closed?
gpshead commentedon May 1, 2022
Hah, joy, but not surprised, that at least proves that failsafes knobs to let modern things be disabled have uses.
The 3.10 PR being merged should close this issue out.
[3.10] gh-91401: Conservative backport of `subprocess._USE_VFORK` (#9…
Remove subprocess._USE_VFORK escape hatch
Remove subprocess._USE_VFORK escape hatch
gh-121381 Remove subprocess._USE_VFORK escape hatch (#121383)
pythongh-121381 Remove subprocess._USE_VFORK escape hatch (python#121383
arthur-proglove commentedon Sep 10, 2024
Hi,
Sorry to dig up this old issue. We are trying to figure out a kernel crash in our Python application and figure out it's always coming from sys_vfork.
Python: 3.11.2
System: Debian 12 on a Raspberry Pi Zero 2W
Kernel: v6.6.47
I have not tried to unset
_USE_VFORKbut maybe it could be related?vstinner commentedon Sep 10, 2024
This issue is closed. I suggest you to open a new issue.