Multiproc plugin with numpy 1.9.0

[divetea]

Summary

I am trying to use the Multiproc plugin but it does not send the tasks to the workers. I traced the reason for this to a numpy expression in _send_procs_to_workers():
jobids = np.nonzero(~self.proc_done & (self.depidx.sum(0) == 0))[1]
in https://github.com/nipy/nipype/blob/master/nipype/pipeline/plugins/multiproc.py (line 216)

Actual behavior

jobids is of type numpy.matrixlib.defmatrix.matrix with shape: (1,1)

Expected behavior

jobids should be of type numpy.ndarray with shape (1,)

I can achieve this behavior with:
jobids = np.asarray(np.nonzero(~self.proc_done & (self.depidx.sum(0) == 0))[1]).flatten()

Then everything is fine and works for all my pipelines.

How to replicate the behavior

I am using numpy version 1.9.0 which is still matching nipype's requirements, isn't it?

Platform details:

{'nibabel_version': '2.1.0', 'sys_executable': '/usr/bin/python', 'networkx_version': '1.11', 'numpy_version': '1.9.0', 'sys_platform': 'linux2', 'sys_version': '2.7.9 (default, Jun 29 2016, 13:08:31) \n[GCC 4.9.2]', 'commit_source': 'repository', 'commit_hash': '8384523', 'pkg_path': '/home/klipsy1.psych.tu-dresden.de/foerster/nipype/nipype', 'nipype_version': '0.14.1-dev+g8384523', 'traits_version': '4.6.0', 'scipy_version': '0.14.0'}
0.14.1-dev+g8384523

[divetea]

There is a reference to a discussion here: #2200 (comment)
And I think my problem is what he meant by:
"But it's a little less explicit that depidx is a thing that is not an ndarray and we need to be careful about how it interacts with a vector like proc_done." : @effigies

[divetea]

The first commit that introduced this problem was 6402981 by @oesteban but a lot has changed since then so I'm not sure how relevant this commit is.

[mgxd]

@divetea thanks for this report - could you provide a small working example? It would be nice to have a test in place to catch this in the future

[divetea]

I am not sure what a small working example would look like, I guess any workflow will do that results in a call of '_send_procs_to_workers'. It depends more on the environment ... I assume it has to do with our "old" numpy version which we have to keep for compatibility with other stuff, but I am not sure. Maybe someone can confirm this on a newer system. I can talk to my boss if i can invest some time in a proper test but otherwise I don't have time at the moment -> student + exam period.
It would be great if someone actually pushed this, I can't go through the whole procedure right now since installing the developer mode and testing is hard on a system where i don't have a lot of rights. And I don't even know how to add the proper tag to this issue ;)
Also I don't know if this is a beautiful enough solution since it's only a quick workaround. Can someone with numpy knowledge have a look at this?
Thank you very much, keep up the good work.
Cheers

[effigies]

Maybe simplest is to revert to the original:

jobids = np.flatnonzero(~self.proc_done & (self.depidx.sum(axis=0) == 0).__array__())

It's ugly but explicit. @divetea can you confirm this also fixes your problem?

[divetea]

@effigies Yes i can confirm that this fixes the Issue! Thank you