Skip to content

Refactoring of POSIX process logic #208

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jul 12, 2021
Merged

Conversation

bgamari
Copy link
Contributor

@bgamari bgamari commented Jun 28, 2021

This is a complete rewrite of posix/runProcess.c. There are a few goals
of this rewrite:

  • fix a long-standing and serious bug in the execvpe fallback path, which
    uses non-reentrant functions after forking. This is of course undefined
    behavior and has been causing failures under Darwin's Rosetta binary
    translation engine (see GHC #19994).

  • eliminate code duplication in the fork/exec implementation.

  • introduce support for posix_spawn, allowing us to unload a significant
    amount of complexity in some cases. This is particularly desireable as the
    cost of fork has increased considerably in some cases on recent Darwin
    releases (namely when MAP_JIT mappings are used; see 1)

While posix_spawn is often a win, there are unfortunately several cases where
it cannot be used:

  • posix_spawn_file_actions_addchdir_np is broken on Darwin

  • POSIX_SPAWN_SETSID is only supported on mac 10.15 and later, but doesn't
    return a proper error code when not supported

  • the originally-specified semantics of posix_spawn_file_actions_adddup2 are
    unsafe and have been amended (see 3) but not all implementations have
    caught up (musl has 4, glibc did later 5, Darwin seemingly hasn't) there appears
    to be no support at all for setuid and setgid

  • spawn is significantly slower than fork on some Darwin releases (see 6)

To address this we first try using posix_spawn, falling back on fork/exec
if we encounter a case which the former cannot handle.

@bgamari bgamari changed the title Refactoring of POSIX process logic WIP: Refactoring of POSIX process logic Jun 28, 2021
Copy link
Collaborator

@snoyberg snoyberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a large enough change in code that I'm honestly not that familiar that I can't offer a meaningful review. With tests passing, and based on you as the author, I feel comfortable merging it. I'd probably also pull this into typed-process and run the test suite there as well. I've added a suggestion to fix the CI failure, and then I'll do a little more testing.

bgamari added 4 commits July 5, 2021 17:04
Previously this was a separate argument despite the fact that we already
pass a flags argument.
This is a complete rewrite `posix/runProcess.c`. There are a few goals
of this rewrite:

 * fix a long-standing and serious bug in the `execvpe` fallback path, which
   uses non-reentrant functions after `fork`ing. This is of course undefined
   behavior and has been causing failures under Darwin's Rosetta binary
   translation engine (see GHC #19994).

 * eliminate code duplication in the `fork/exec` implementation.

 * introduce support for `posix_spawn`, allowing us to unload a significant
   amount of complexity in some cases. This is particularly desireable as the
   cost of `fork` has increased considerably in some cases on recent Darwin
   releases (namely when `MAP_JIT` mappings are used; see [1])

While `posix_spawn` is often a win, there are unfortunately several cases where
it cannot be used:

 * `posix_spawn_file_actions_addchdir_np` is broken on Darwin

 * `POSIX_SPAWN_SETSID` is only supported on mac 10.15 and later, but doesn't
   return a proper error code when not supported

 * the originally-specified semantics of `posix_spawn_file_actions_adddup2` are
   unsafe and have been amended (see [3]) but not all implementations have
   caught up (musl has [4], glibc did later [5], Darwin seemingly hasn't) there appears
   to be no support at all for setuid and setgid

 * `spawn` is significantly slower than fork on some Darwin releases (see [6])

To address this we first try using `posix_spawn`, falling back on `fork/exec`
if we encounter a case which the former cannot handle.

[1]: libuv/libuv#3064
[2]: https://www.austingroupbugs.net/view.php?id=411
[3]: rust-lang/rust#80537
[4]: https://git.musl-libc.org/cgit/musl/commit/?id=6fc6ca1a323bc0b6b9e9cdc8fa72221ae18fe206
[5]: https://sourceware.org/bugzilla/show_bug.cgi?id=23640
[6]: https://discuss.python.org/t/multiprocessing-spawn-default-on-macos-since-python-3-8-is-slower-than-fork-method/5910/4
@bgamari
Copy link
Contributor Author

bgamari commented Jul 12, 2021

Hi Michael, I believe this is now ready for further testing if you are able. Otherwise, I have also done a fair amount of testing locally (on Darwin and Linux) and I believe the work is sound.

@bgamari bgamari changed the title WIP: Refactoring of POSIX process logic efactoring of POSIX process logic Jul 12, 2021
@bgamari bgamari changed the title efactoring of POSIX process logic Refactoring of POSIX process logic Jul 12, 2021
@snoyberg
Copy link
Collaborator

I've done additional testing on both Windows and Linux, by running both the local test suite and the typed-process test suite. Both are now passing. Merging now, thanks! Do you want me to make a release at this point?

@snoyberg snoyberg merged commit 9410f77 into haskell:master Jul 12, 2021
@bgamari
Copy link
Contributor Author

bgamari commented Jul 12, 2021

Thanks, @snoyberg! I've opened #209 with a small fix and a minor refactor. Otherwise I think we are ready for a release. I think we will use this release in 8.10.6, 9.0.2, and 9.2.1

snoyberg added a commit that referenced this pull request Jul 13, 2021
@snoyberg
Copy link
Collaborator

Cool, release 1.6.13.0 made including the changes from #209.

@Bodigrim
Copy link
Contributor

Builds fail for 1.6.13.0 with

cbits/posix/find_executable.c:12:20: error:
     fatal error: common.h: No such file or directory
   |
12 | #include "common.h"

https://matrix.hackage.haskell.org/#/package/process/1.6.13.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants