Freeze support documentation is misleading.

[SridharIyer]

BPO	36153
Nosy	@pitrou, @applio, @Windsooon, @samschott
Files	run_server_min.py: Python code to reproduce the issue.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-03-01.00:49:32.769>
labels = ['3.7', 'docs']
title = 'Freeze support documentation is misleading.'
updated_at = <Date 2021-05-17.12:10:25.917>
user = 'https://bugs.python.org/SridharIyer'

bugs.python.org fields:

activity = <Date 2021-05-17.12:10:25.917>
actor = 'samschott'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2019-03-01.00:49:32.769>
creator = 'Sridhar Iyer'
dependencies = []
files = ['48182']
hgrepos = []
issue_num = 36153
keywords = []
message_count = 4.0
messages = ['336881', '336885', '336932', '393809']
nosy_count = 6.0
nosy_names = ['pitrou', 'docs@python', 'davin', 'Windson Yang', 'Sridhar Iyer', 'samschott']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue36153'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5', 'Python 3.6', 'Python 3.7']

Linked PRs

• gh-80334: fix multiprocessing.freeze_support for other spawn platforms #134462
• GH-80334: make multiprocessing freeze_support() usable on all spawn default platforms, not just windows #134617
• [3.14] gh-80334: fix multiprocessing.freeze_support for other spawn platforms (GH-134462) #134619
• [3.13] gh-80334: fix multiprocessing.freeze_support for other spawn platforms (GH-134462) #134620
• [3.14] Revert "gh-80334: fix multiprocessing.freeze_support for other spawn platforms (GH-134462) (#134619) #134658

[SridharIyer]

The documentation on freeze_support listed on https://docs.python.org/3/library/multiprocessing.html need to be fixed

This mentions:
"Calling freeze_support() has no effect when invoked on any operating system other than Windows. In addition, if the module is being run normally by the Python interpreter on Windows (the program has not been frozen), then freeze_support() has no effect."

This is not true. Sklearn/tensorflow libraries tend to cause an infinite loop when frozen with pyinstaller (tested on python 3.6 on ubuntu 14.04). freeze_support is the only way to get around the situation and should be included before including any other module that includes a multiprocessing library (not just in main).

[Windsooon]

IIUC, your script (using Sklearn/tensorflow) will cause an infinite loop when building with pyinstaller. Would you mind upload an example script so I can try to reproduce it?

[SridharIyer]

Please find the attached python file where the issue is seen.
The cli to create an executable was:
$pyinstaller run_server_min.spec

Here are the contents of the file (this doesn't support multiple file uploads):
================
# -- mode: python ; coding: utf-8 --

block_cipher = None


a = Analysis(['run_server_min.py'],
             pathex=['<PUT ABSOLUTE PATH OF ABOVE FILE HERE>'],
             binaries=[],
             datas=[],
             hiddenimports=['sklearn.neighbors.typedefs', 'sklearn.neighbors.quad_tree', 'sklearn.tree._utils', 'xgboost', 'xgboost.libpath'],
             hookspath=['pyhooks'],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          a.binaries,
          a.zipfiles,
          a.datas,
          [],
          name='run_server_min',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=False,
          runtime_tmpdir=None,
          console=True )

=========

$ pyinstaller run_server_min.spec
69 INFO: PyInstaller: 3.5.dev0+cb8d10af6
69 INFO: Python: 3.6.7
70 INFO: Platform: Linux-3.16.0-77-generic-x86_64-with-debian-jessie-sid
...

When your run ./dist/run_server_min that is generated, it'll spawn the process multiple times. The issue goes away when you add freeze_support on the top.

[samschott]

I think freeze_support becomes relevant always when the start method is spawn since sys.executable will point to the frozen app bundle instead of the Python interpreter. Spawn was historically only used on Windows (therefore the note in the docs) but now also is the default for macOS. The start method can of course also be set manually and it might therefore make more sense to specify the start method instead of the platform in the docs.

[mssalvatore]

I think freeze_support becomes relevant always when the start method is spawn

This is consistent with my experience.

it might therefore make more sense to specify the start method instead of the platform in the docs.

+1

[uwu-420]

I also had to use multiprocessing.freeze_support for an app created with pyinstaller on macOS that uses mlserver. I guess it would be good to update the docs here.

[eddymul]

Do we need to update

cpython/Lib/multiprocessing/context.py

Lines 144 to 150 in 11d77d2

	def freeze_support(self):
	'''Check whether this is a fake forked process in a frozen executable.
	If so then run code specified by commandline and exit.
	'''
	if sys.platform == 'win32' and getattr(sys, 'frozen', False):
	from .spawn import freeze_support
	freeze_support()

to check for spawn_method instead of sys.platform?