MulitProc Workflows locking up

[blakedewey]

I think this may be a result of #1689. When I run a large MultiProc workflow, it runs for one or two nodes, then hangs. There is nothing reported and the nodes have finished successfully. I am also unable to use SIGTERM to end the process and have to manually close the terminal window. Moving back before this change seems to fix the issue.

[satra]

@carolFrohlich - could you please take a look at this?

[chrisgorgo]

CCing @ccraddock who promised to help. We have experienced those as well. Deadlocks are stochastic in nature so there are no steps to reproduce. Here are nipype tests which timed out due to deadlocks in multiproc:

https://circleci.com/gh/nipy/nipype/1690
https://circleci.com/gh/nipy/nipype/1686
https://circleci.com/gh/nipy/nipype/1686
https://circleci.com/gh/nipy/nipype/1685
https://circleci.com/gh/nipy/nipype/1682
https://circleci.com/gh/nipy/nipype/1676
https://circleci.com/gh/nipy/nipype/1674

the problem seems to affect Python 2.7 more often than python 3.5

[chrisgorgo]

It seems to be affecting travis tests as well:

https://api.travis-ci.org/jobs/180558248/log.txt?deansi=true

nipype.pipeline.plugins.tests.test_linear.test_run_in_series([1, 1], [1, 1]) ... ok


No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

The build has been terminated

On a successful run the next test after test_run_in_series is nipype.pipeline.plugins.tests.test_multiproc.test_run_multiproc

[djarecka]

As far as I remember my problems with travis were always in the second run only (python2.7; INSTALL_DEB_DEPENDECIES=false), e.g. https://travis-ci.org/djarecka/nipype/jobs/180111821
Checking now my CircleCI builds I also can't find problems with python3.

[jvarada]

Is there a reason for using the multiprocessing library and handling locks for the main thread manually rather than using concurrent.futures (or futures in py2.7), sending threads out as Futures, and waiting on the Futures instead?

[chrisgorgo]

@ccraddock is asking for example pipelines that experienced this problem. Please add to the list.

The fmri_spm_nested pipeline included in nipype has experienced this issue at least a couple of times: https://circleci.com/gh/nipy/nipype/1690
https://circleci.com/gh/nipy/nipype/1674

We have seen it while running fmriprep as well (before switching to Python 3).

Unfortunately this issue seems to be stochastic in nature and is not easily reproducible.

I hope this helps

[satra]

@jvarada - mostly historical and our older support for 2.6. this would be a good thing to reconsider as we rewrite the engine.

[djarecka]

Not sure if you're testing on OSX, but I've just run locally and have problem with test_multiproc.py, python2.7.

@chrisfilo did you have any specific reason why you made skipif using 'TRAVIS_PYTHON_VERSION', i.e. os.environ.get('TRAVIS_PYTHON_VERSION') == '2.7'? Is it ok to change this using sys.version_info? Just trying to eliminate this problem when running make tests