-
-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
py.test
command is not compatible with multiprocessing
start-methods other than *fork*
#958
Comments
py.test
command is not compatible with multiprocessing
modulepy.test
command is not compatible with multiprocessing
start-methods other than fork
py.test
command is not compatible with multiprocessing
start-methods other than fork
py.test
command is not compatible with multiprocessing
start-methods other than *fork*
that's an issue with setuptools/pip as well as multiprocessing - py.test does not make the scripts but use the entry-points feature that existed since way before multiprocessing also the script should differ these days for 3 main reasons a) pip creates a different script for wheels (recent releases have those) from my POV this is nothing py.test should fix on its own, this should be fixed in pip/setuptools because proper script creation on win32 is a major pita, at least personally i don't want to be responsible for it |
Is there anything that can be done by |
Workaround might be something like |
- added tests for user class, multiple processes, feedback, and assignment type. - added script for running the example. - updated documentation on running the tests. - update documentation on running the example.
The python multiprocessing library doesn't play very nicely when creating Pools after initializing complex state (e.g. the KuduClient). Because of how it forks, the library may even copy lock states, and lead to odd situations, like multiple threads waiting on a lock that isn't held by any thread in the process. This was the case in KUDU-2686, which resulted in a hang in test_scantoken.py. Upon inspection[1], there appeared to be multiple threads in a process waiting on the same sys_futex, but none of them held it. [2] tips some pieces to make it more reproducible (tested on Ubuntu 14.04, where the issue was first reported). Starting with python3.4, there is supposedly a way around this, which is to use a different process-startup pattern, though this isn't available in python2. This patch removes usage of multiprocessing entirely. While it can be argued that multiprocessing provides extra test coverage, given that multiprocessing is known to have such issues[3][4], that this isn't the first time we've been bitten by this forking issue, that it's test-only, and that scan tokens are tested in the C++ client as well, "dumbing" down the test doesn't seem unreasonable. [1] https://gist.github.com/andrwng/d2d21c551362ddd564926c2a4ec406ae [2] https://gist.github.com/andrwng/cc6c211c62b1235cc58944d513ba6655 [3] pytest-dev/pytest#958 [4] https://codewithoutrules.com/2018/09/04/python-multiprocessing/ Change-Id: Ia9aa91191d54801731da27e5f132b3c96af0efa1 Reviewed-on: http://gerrit.cloudera.org:8080/12494 Tested-by: Kudu Jenkins Reviewed-by: Jordan Birdsell <jtbirdsell@apache.org> Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
The python multiprocessing library doesn't play very nicely when creating Pools after initializing complex state (e.g. the KuduClient). Because of how it forks, the library may even copy lock states, and lead to odd situations, like multiple threads waiting on a lock that isn't held by any thread in the process. This was the case in KUDU-2686, which resulted in a hang in test_scantoken.py. Upon inspection[1], there appeared to be multiple threads in a process waiting on the same sys_futex, but none of them held it. [2] tips some pieces to make it more reproducible (tested on Ubuntu 14.04, where the issue was first reported). Starting with python3.4, there is supposedly a way around this, which is to use a different process-startup pattern, though this isn't available in python2. This patch removes usage of multiprocessing entirely. While it can be argued that multiprocessing provides extra test coverage, given that multiprocessing is known to have such issues[3][4], that this isn't the first time we've been bitten by this forking issue, that it's test-only, and that scan tokens are tested in the C++ client as well, "dumbing" down the test doesn't seem unreasonable. [1] https://gist.github.com/andrwng/d2d21c551362ddd564926c2a4ec406ae [2] https://gist.github.com/andrwng/cc6c211c62b1235cc58944d513ba6655 [3] pytest-dev/pytest#958 [4] https://codewithoutrules.com/2018/09/04/python-multiprocessing/ Change-Id: Ia9aa91191d54801731da27e5f132b3c96af0efa1 Reviewed-on: http://gerrit.cloudera.org:8080/12494 Tested-by: Kudu Jenkins Reviewed-by: Jordan Birdsell <jtbirdsell@apache.org> Reviewed-by: Alexey Serbin <aserbin@cloudera.com> Reviewed-on: http://gerrit.cloudera.org:8080/12515 Reviewed-by: Grant Henke <granthenke@apache.org>
The python multiprocessing library doesn't play very nicely when creating Pools after initializing complex state (e.g. the KuduClient). Because of how it forks, the library may even copy lock states, and lead to odd situations, like multiple threads waiting on a lock that isn't held by any thread in the process. This was the case in KUDU-2686, which resulted in a hang in test_scantoken.py. Upon inspection[1], there appeared to be multiple threads in a process waiting on the same sys_futex, but none of them held it. [2] tips some pieces to make it more reproducible (tested on Ubuntu 14.04, where the issue was first reported). Starting with python3.4, there is supposedly a way around this, which is to use a different process-startup pattern, though this isn't available in python2. This patch removes usage of multiprocessing entirely. While it can be argued that multiprocessing provides extra test coverage, given that multiprocessing is known to have such issues[3][4], that this isn't the first time we've been bitten by this forking issue, that it's test-only, and that scan tokens are tested in the C++ client as well, "dumbing" down the test doesn't seem unreasonable. [1] https://gist.github.com/andrwng/d2d21c551362ddd564926c2a4ec406ae [2] https://gist.github.com/andrwng/cc6c211c62b1235cc58944d513ba6655 [3] pytest-dev/pytest#958 [4] https://codewithoutrules.com/2018/09/04/python-multiprocessing/ Change-Id: Ia9aa91191d54801731da27e5f132b3c96af0efa1 Reviewed-on: http://gerrit.cloudera.org:8080/12494 Tested-by: Kudu Jenkins Reviewed-by: Jordan Birdsell <jtbirdsell@apache.org> Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
The pytest framework seems to have its troubles with multiprocessing. Prior to this commit, running `test_task.py` individually would succeed, while running the whole test suite would fail, just as one example. This commit addresses those issues by wrapping `pytest.main` execution into the `if __name__ == '__main__'` guard and ensuring that the utopya module is in the system path (regardless of how pytest may have modified it). See pytest-dev/pytest#958 for more info.
The pytest framework seems to have its troubles with multiprocessing. Prior to this commit, running `test_task.py` individually would succeed, while running the whole test suite would fail, just as one example. This commit addresses those issues by wrapping `pytest.main` execution into the `if __name__ == '__main__'` guard and ensuring that the utopya module is in the system path (regardless of how pytest may have modified it). See pytest-dev/pytest#958 for more info.
Currently, it is not possible to use the
py.test
command to run tests that use themultiprocessing
module with a start-method other than fork. This is because thepy.test
script generated by setuptools does not include aif __name__ == '__main__'
guard:This means that as soon as a test tries to spawn a new process, this process starts executing the test-suite itself (the
-s
option is necessary to see the output of the second process):This is easily rectified by using a custom
py.test
command:But it would be much nicer if the standard
py.test
command could be used.The text was updated successfully, but these errors were encountered: