Handle a non-importable __main__ in multiprocessing

[ogrisel]

BPO	19946
Nosy	@brettcannon, @ncoghlan, @pitrou, @larryhastings, @tiran, @ericsnowcurrently, @zware, @ogrisel
Files	issue19946.diff issue19946_pep_451_multiprocessing_v2.diff: Work in progress patch (Fork+, ForkServer-, Spawn--) test_multiprocessing_main_handling.py: Final test case issue19946.diff skip_forkserver.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2013-12-20.14:33:20.194>
created_at = <Date 2013-12-10.13:08:59.356>
labels = ['type-bug', 'library', 'release-blocker']
title = 'Handle a non-importable __main__ in multiprocessing'
updated_at = <Date 2015-02-24.09:59:58.484>
user = 'https://github.com/ogrisel'

bugs.python.org fields:

activity = <Date 2015-02-24.09:59:58.484>
actor = 'ncoghlan'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2013-12-20.14:33:20.194>
closer = 'ncoghlan'
components = ['Library (Lib)']
creation = <Date 2013-12-10.13:08:59.356>
creator = 'Olivier.Grisel'
dependencies = []
files = ['33091', '33162', '33175', '33201', '33222']
hgrepos = []
issue_num = 19946
keywords = ['patch']
message_count = 47.0
messages = ['205810', '205831', '205832', '205833', '205847', '205854', '205857', '205885', '205893', '205894', '205895', '205898', '205904', '206115', '206118', '206120', '206126', '206129', '206133', '206134', '206137', '206138', '206140', '206180', '206224', '206228', '206229', '206230', '206252', '206260', '206296', '206297', '206298', '206301', '206421', '206424', '206478', '206559', '206575', '206607', '206650', '206653', '206655', '206659', '206677', '206678', '206683']
nosy_count = 10.0
nosy_names = ['brett.cannon', 'ncoghlan', 'pitrou', 'larry', 'christian.heimes', 'python-dev', 'sbt', 'eric.snow', 'zach.ware', 'Olivier.Grisel']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue19946'
versions = ['Python 3.4']

[ogrisel]

Here is a simple python program that uses the new forkserver feature introduced in 3.4b1:

name: checkforkserver.py
"""
import multiprocessing
import os

def do(i):
    print(i, os.getpid())


def test_forkserver():
    mp = multiprocessing.get_context('forkserver')
    mp.Pool(2).map(do, range(3))


if __name__ == "__main__":
    test_forkserver()
"""

When running this using the "python check_forkserver.py" command everything works as expected.

When running this using the nosetests launcher ("nosetests -s check_forkserver.py"), I get:

"""
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/Python-HEAD/lib/python3.4/multiprocessing/forkserver.py", line 141, in main
    spawn.import_main_path(main_path)
  File "/opt/Python-HEAD/lib/python3.4/multiprocessing/spawn.py", line 252, in import_main_path
    methods.init_module_attrs(main_module)
  File "<frozen importlib._bootstrap>", line 1051, in init_module_attrs
AttributeError: 'NoneType' object has no attribute 'loader'
"""

Indeed, the spec variable in multiprocessing/spawn.py's import_main_path
function is None as the nosetests script is not a regular python module: in particular is does not have a ".py" extension.

If I copy or symlink or renamed the "nosetests" script as "nosetests.py" in the same folder, this works as expected. I am not familiar enough with the importlib machinery to suggest a fix for this bug.

Also there is a typo in the comment: "causing a psuedo fork bomb" => "causing a pseudo fork bomb".

Note: I am running CPython head updated today.

[pitrou]

This sounds related to the ModuleSpec changes.

[brettcannon]

The returning of None means that importlib.find_spec() didn't find the spec/loader for the specified module. So the question is exactly what module is being passed to importlib.find_spec() and why isn't it finding a spec/loader for that module.

Did this code work in Python 3.3?

[brettcannon]

I see Oliver says he is testing a new forkserver feature from 3.4b1, so it might not necessarily be importlib's fault then. Does using the old importlib.find_loader() approach work?

[ogrisel]

So the question is exactly what module is being passed to importlib.find_spec() and why isn't it finding a spec/loader for that module.

The module is the nosetests python script. module_name == 'nosetests' in this case. However, nosetests is not considered an importable module because of the missing '.py' extension in the filename.

Did this code work in Python 3.3?

This code did not exist in Python 3.3.

[brettcannon]

So at the bare minimum, the multiprocessing code should raise an ImportError when it can't find the spec for the module to help debug this kind of thing. Also that typo should get fixed.

Second, there is no way that 'nosetests' will ever succeed as an import since, as Oliver pointed out, it doesn't end in '.py' or any other identifiable way for a finder to know it can handle the file. So this is not a bug and simply a side-effect of how import works. The only way around it would be to symlink nosetests to nosetests.py or to somehow pre-emptively set up 'nosetests' for supported importing.

[sbt]

I guess this is a case where we should not be trying to import the main module. The code for determining the path of the main module (if any) is rather crufty.

What is sys.modules['__main__'] and sys.modules['__main__'].__file__ if you run under nose?

[ericsnowcurrently]

There is definitely room for improvement relative to module specs and __main__ (that's the topic of issue bpo-19701). That issue is waiting for __main__ to get a proper spec (see issues bpo-19700 and bpo-19697).

[ogrisel]

I agree that a failure to lookup the module should raise an explicit exception.

Second, there is no way that 'nosetests' will ever succeed as an import since, as Oliver pointed out, it doesn't end in '.py' or any other identifiable way for a finder to know it can handle the file. So this is not a bug and simply a side-effect of how import works. The only way around it would be to symlink nosetests to nosetests.py or to somehow pre-emptively set up 'nosetests' for supported importing.

I don't agree that (unix) Python programs that don't end with ".py" should be modified to have multiprocessing work correctly. I think it should be the multiprocessing responsibility to transparently find out how to spawn the new process independently of the fact that the program ends in '.py' or not.

Note: the fork mode works always under unix (with or without the ".py" extension). The spawn mode always work under windows as AFAIK there is no way to have Python programs that don't end in .py under windows and furthermore I think multiprocessing does execute the __main__ under windows (but I haven't tested if it's still the case in Python HEAD).