multiprocessing freeze_support needed outside win32

[dancol]

BPO	32146
Nosy	@pfmoore, @ronaldoussoren, @pitrou, @tjguk, @zware, @zooba, @applio, @bbayles, @miss-islington, @orn688
PRs	bpo-32146: multiprocessing freeze_support needed outside win32 #5195 bpo-32146: Add documentation about frozen executables on Unix #5850 [3.7] bpo-32146: Add documentation about frozen executables on Unix (GH-5850) #11511 [3.7] bpo-32146: Add documentation about frozen executables on Unix (GH-5850) #11511 [3.7] bpo-32146: Add documentation about frozen executables on Unix (GH-5850) #11511

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-11-27.10:33:55.596>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = 'multiprocessing freeze_support needed outside win32'
updated_at = <Date 2021-03-29.12:34:42.683>
user = 'https://bugs.python.org/dancol'

bugs.python.org fields:

activity = <Date 2021-03-29.12:34:42.683>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-11-27.10:33:55.596>
creator = 'dancol'
dependencies = []
files = []
hgrepos = []
issue_num = 32146
keywords = ['patch']
message_count = 11.0
messages = ['307043', '307527', '309886', '311274', '333411', '333412', '333414', '333432', '333460', '333533', '334201']
nosy_count = 11.0
nosy_names = ['paul.moore', 'ronaldoussoren', 'pitrou', 'tim.golden', 'zach.ware', 'steve.dower', 'dancol', 'davin', 'bbayles', 'miss-islington', 'onew']
pr_nums = ['5195', '5850', '11511', '11511', '11511']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32146'
versions = ['Python 3.7', 'Python 3.8']

Linked PRs

• gh-76327: freeze_support for non-Windows platforms #104607

[dancol]

multiprocessing's freeze_support makes freshly-launched subprocesses integrate correctly until the multiprocessing ecosystem even when the main executable is some application binary instead of a Python interpreter. The documentation and code assume that this support is needed only on win32, but it's equally applicable to POSIX systems using the spawn strategy in combination with systems like cx_freeze.

In particular:

1. The special case in context.py's version of freeze_support() for win32 should be removed

2. The semaphore tracker should correctly tickle the freeze support

3. The documentation should be updated to be platform-neutral

[pitrou]

Thank you for reporting this. Would you like to submit a PR?

[bbayles]

I ran into this issue, and found references to it on StackOverflow [1] and GitHub [2] as well. I found that the problem applies to both the 'spawn' and 'forkserver' start methods on Linux.

I made an attempt to implement dancol's fix above. (1) and (3) are straightforward, but if there's an elegant way to do the "tickle" in (2) I wasn't able to figure it out?

My branch [3] has something that seems to work with the 'spawn' method and cx_Freeze. Perhaps someone could look at that and tell me if I'm on a reasonable track? If so, I can attempt to extend the method to cover 'forkserver' as well.

[1] https://stackoverflow.com/q/47325297/353839

[2] marcelotduarte/cx_Freeze#264

[3] master...bbayles:bpo-32146-freeze_support

[bbayles]

I presume it's too late to get my pull request in for 3.7, but perhaps we could get a note in the documentation for this before release?

That could be as simple as:

.. warning::

The ``'spawn'`` and ``'forkserver'`` start methods cannot currently be used with "frozen" executables (i.e., binaries produced by packages like **PyInstaller** and **cx_Freeze**) on Unix. The ``'fork'`` start method does work.

I will make that into a PR if it seems like a good option.

[pitrou]

I'm sorry about not seeing your PR before. Would you like to update it against current git master?

(note I'm not thrilled by the use of the "ast" module; I hope we can change the command-line args so that this isn't needed)

[vstinner]

New changeset bab4bbb by Victor Stinner (Bo Bayles) in branch 'master':
bpo-32146: Add documentation about frozen executables on Unix (GH-5850)
bab4bbb

[miss-islington]

New changeset b9cd38f by Miss Islington (bot) in branch '3.7':
bpo-32146: Add documentation about frozen executables on Unix (GH-5850)
b9cd38f

[bbayles]

Thanks for the note. I've merged in master and fixed a conflict in the test file.

In an earlier rev I tried to do the argument parsing without ast.literal_eval, but found it awkward to support all the ways [1] can manifest. You might have a better idea?

[1] https://github.com/python/cpython/blob/3.7/Lib/multiprocessing/forkserver.py#L106-L107

[pitrou]

The key here would be to replace function execution with module execution and command-line arguments.

So instead of:

python -c 'from multiprocessing.forkserver import main; main(ARG1, ARG2...)'

you would do:

python -m multiprocessing.forkserver ARG1 ARG2 ....

Then it should become easy to extract the concatenated command lines.

(and if that works, please do the same for semaphore_tracker ;-))