Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

py.test command is not compatible with multiprocessing start-methods other than *fork* #958

Closed
Nikratio opened this issue Aug 25, 2015 · 4 comments

Comments

@Nikratio
Copy link
Contributor

Currently, it is not possible to use the py.test command to run tests that use the multiprocessing module with a start-method other than fork. This is because the py.test script generated by setuptools does not include a if __name__ == '__main__' guard:

$ cat /usr/bin/py.test-3
#! /usr/bin/python3

import sys
from pkg_resources import load_entry_point
sys.exit(load_entry_point('pytest==2.6.3', 'console_scripts', 'py.test')())

This means that as soon as a test tries to spawn a new process, this process starts executing the test-suite itself (the -s option is necessary to see the output of the second process):

$ py.test-3 test_mp.py -s
============================================== test session starts ==============================================
platform linux -- Python 3.4.2 -- py-1.4.25 -- pytest-2.6.3
collected 1 items 

test_mp.py ============================================== test session starts ==============================================
platform linux -- Python 3.4.2 -- py-1.4.25 -- pytest-2.6.3
collected 1 items 

test_mp.py F
[...]

This is easily rectified by using a custom py.test command:

$ cat run_test.py 
#!/usr/bin/python3
import pytest
import sys
if __name__ == '__main__':
    sys.exit(pytest.main([__file__] + sys.argv[1:]))
$ python3 run_test.py test_mp.py  -s
============================================== test session starts ==============================================
platform linux -- Python 3.4.2 -- py-1.4.25 -- pytest-2.6.3
collected 1 items 

test_mp.py hello
.

=========================================== 1 passed in 0.11 seconds ============================================

But it would be much nicer if the standard py.test command could be used.

@Nikratio Nikratio changed the title py.test command is not compatible with multiprocessing module py.test command is not compatible with multiprocessing start-methods other than fork Aug 25, 2015
@Nikratio Nikratio changed the title py.test command is not compatible with multiprocessing start-methods other than fork py.test command is not compatible with multiprocessing start-methods other than *fork* Aug 25, 2015
@RonnyPfannschmidt
Copy link
Member

that's an issue with setuptools/pip as well as multiprocessing - py.test does not make the scripts but use the entry-points feature that existed since way before multiprocessing

also the script should differ these days for 3 main reasons

a) pip creates a different script for wheels (recent releases have those)
b) there is .exe file support on windows
c) pip >7 caches wheels whenever it can

from my POV this is nothing py.test should fix on its own, this should be fixed in pip/setuptools

because proper script creation on win32 is a major pita, at least personally i don't want to be responsible for it

@nicoddemus
Copy link
Member

Is there anything that can be done by py.test at all here? From @RonnyPfannschmidt's, explanation I gather that nothing can be done and we should close this. Is this correct?

@RonnyPfannschmidt
Copy link
Member

Workaround might be something like pyhon -m pytest

@Nikratio
Copy link
Contributor Author

lmz referenced this issue in Fast-Trips/fast-trips May 8, 2017
 - added tests for user class, multiple processes, feedback, and
assignment type.
 - added script for running the example.
 - updated documentation on running the tests.
 - update documentation on running the example.
e-lo added a commit to Fast-Trips/fast-trips that referenced this issue May 9, 2017
asfgit pushed a commit to apache/kudu that referenced this issue Feb 15, 2019
The python multiprocessing library doesn't play very nicely when
creating Pools after initializing complex state (e.g. the KuduClient).
Because of how it forks, the library may even copy lock states, and lead
to odd situations, like multiple threads waiting on a lock that isn't
held by any thread in the process.

This was the case in KUDU-2686, which resulted in a hang in
test_scantoken.py. Upon inspection[1], there appeared to be multiple
threads in a process waiting on the same sys_futex, but none of them
held it. [2] tips some pieces to make it more reproducible (tested on
Ubuntu 14.04, where the issue was first reported).

Starting with python3.4, there is supposedly a way around this, which is
to use a different process-startup pattern, though this isn't available
in python2.

This patch removes usage of multiprocessing entirely. While it can be
argued that multiprocessing provides extra test coverage, given that
multiprocessing is known to have such issues[3][4], that this isn't the
first time we've been bitten by this forking issue, that it's test-only,
and that scan tokens are tested in the C++ client as well, "dumbing"
down the test doesn't seem unreasonable.

[1] https://gist.github.com/andrwng/d2d21c551362ddd564926c2a4ec406ae
[2] https://gist.github.com/andrwng/cc6c211c62b1235cc58944d513ba6655
[3] pytest-dev/pytest#958
[4] https://codewithoutrules.com/2018/09/04/python-multiprocessing/

Change-Id: Ia9aa91191d54801731da27e5f132b3c96af0efa1
Reviewed-on: http://gerrit.cloudera.org:8080/12494
Tested-by: Kudu Jenkins
Reviewed-by: Jordan Birdsell <jtbirdsell@apache.org>
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
asfgit pushed a commit to apache/kudu that referenced this issue Feb 19, 2019
The python multiprocessing library doesn't play very nicely when
creating Pools after initializing complex state (e.g. the KuduClient).
Because of how it forks, the library may even copy lock states, and lead
to odd situations, like multiple threads waiting on a lock that isn't
held by any thread in the process.

This was the case in KUDU-2686, which resulted in a hang in
test_scantoken.py. Upon inspection[1], there appeared to be multiple
threads in a process waiting on the same sys_futex, but none of them
held it. [2] tips some pieces to make it more reproducible (tested on
Ubuntu 14.04, where the issue was first reported).

Starting with python3.4, there is supposedly a way around this, which is
to use a different process-startup pattern, though this isn't available
in python2.

This patch removes usage of multiprocessing entirely. While it can be
argued that multiprocessing provides extra test coverage, given that
multiprocessing is known to have such issues[3][4], that this isn't the
first time we've been bitten by this forking issue, that it's test-only,
and that scan tokens are tested in the C++ client as well, "dumbing"
down the test doesn't seem unreasonable.

[1] https://gist.github.com/andrwng/d2d21c551362ddd564926c2a4ec406ae
[2] https://gist.github.com/andrwng/cc6c211c62b1235cc58944d513ba6655
[3] pytest-dev/pytest#958
[4] https://codewithoutrules.com/2018/09/04/python-multiprocessing/

Change-Id: Ia9aa91191d54801731da27e5f132b3c96af0efa1
Reviewed-on: http://gerrit.cloudera.org:8080/12494
Tested-by: Kudu Jenkins
Reviewed-by: Jordan Birdsell <jtbirdsell@apache.org>
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
Reviewed-on: http://gerrit.cloudera.org:8080/12515
Reviewed-by: Grant Henke <granthenke@apache.org>
zhangyifan27 pushed a commit to zhangyifan27/kudu that referenced this issue Feb 28, 2019
The python multiprocessing library doesn't play very nicely when
creating Pools after initializing complex state (e.g. the KuduClient).
Because of how it forks, the library may even copy lock states, and lead
to odd situations, like multiple threads waiting on a lock that isn't
held by any thread in the process.

This was the case in KUDU-2686, which resulted in a hang in
test_scantoken.py. Upon inspection[1], there appeared to be multiple
threads in a process waiting on the same sys_futex, but none of them
held it. [2] tips some pieces to make it more reproducible (tested on
Ubuntu 14.04, where the issue was first reported).

Starting with python3.4, there is supposedly a way around this, which is
to use a different process-startup pattern, though this isn't available
in python2.

This patch removes usage of multiprocessing entirely. While it can be
argued that multiprocessing provides extra test coverage, given that
multiprocessing is known to have such issues[3][4], that this isn't the
first time we've been bitten by this forking issue, that it's test-only,
and that scan tokens are tested in the C++ client as well, "dumbing"
down the test doesn't seem unreasonable.

[1] https://gist.github.com/andrwng/d2d21c551362ddd564926c2a4ec406ae
[2] https://gist.github.com/andrwng/cc6c211c62b1235cc58944d513ba6655
[3] pytest-dev/pytest#958
[4] https://codewithoutrules.com/2018/09/04/python-multiprocessing/

Change-Id: Ia9aa91191d54801731da27e5f132b3c96af0efa1
Reviewed-on: http://gerrit.cloudera.org:8080/12494
Tested-by: Kudu Jenkins
Reviewed-by: Jordan Birdsell <jtbirdsell@apache.org>
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
ThGaskin pushed a commit to utopia-foss/utopia that referenced this issue Dec 22, 2020
The pytest framework seems to have its troubles with multiprocessing.
Prior to this commit, running `test_task.py` individually would succeed,
while running the whole test suite would fail, just as one example.

This commit addresses those issues by wrapping `pytest.main` execution
into the `if __name__ == '__main__'` guard and ensuring that the utopya
module is in the system path (regardless of how pytest may have modified
it). See pytest-dev/pytest#958 for more info.
blsqr added a commit to utopia-foss/utopya that referenced this issue Jul 13, 2022
The pytest framework seems to have its troubles with multiprocessing.
Prior to this commit, running `test_task.py` individually would succeed,
while running the whole test suite would fail, just as one example.

This commit addresses those issues by wrapping `pytest.main` execution
into the `if __name__ == '__main__'` guard and ensuring that the utopya
module is in the system path (regardless of how pytest may have modified
it). See pytest-dev/pytest#958 for more info.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants