Skip to content

FIX: Copy num_threads to MapNode-generated Nodes #2019

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 16, 2017

Conversation

effigies
Copy link
Member

This PR copies the num_threads attribute from source MapNode's interface to those of the generated nodes.

Fixes #2016

@satra
Copy link
Member

satra commented May 14, 2017

lgtm - seems like a simple fix to incorporate in 0.13.1

@effigies
Copy link
Member Author

Actually, I should probably copy the memory estimate, too.

For a test, I can do something like:

mapnode = pe.MapNode(...)
nodes = list(mapnode._make_nodes())
for node in nodes:
    for attr in attrs_we_care_about:
        assert getattr(node, attr) == getattr(mapnode, attr)

Any other candidates for attrs_we_care_about?

@effigies effigies force-pushed the fix/mapnode_parallelism branch from 5787a98 to edfcf84 Compare May 14, 2017 03:35
@@ -1125,6 +1125,8 @@ def _make_nodes(self, cwd=None):
fieldvals = filename_to_list(getattr(self.inputs, field))
logger.debug('setting input %d %s %s', i, field, fieldvals[i])
setattr(node.inputs, field, fieldvals[i])
node._interface.num_threads = self._interface.num_threads
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

quick question, why doesn't the deepcopy a few lines (1115) back copy num_threads?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no idea. But you can run _make_nodes and check for yourself.

I could instead add a BaseInterface.__deepcopy__ function.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In [6]: node = MapNode(BET(), iterfield=['in_file'], name='test')

In [7]: node._interface.num_threads
Out[7]: 1

In [8]: node._interface.num_threads = 5

In [9]: node._interface.num_threads
Out[9]: 5

In [10]: node.interface.num_threads
Out[10]: 5

In [11]: node.interface.num_threads = 4

In [12]: node._interface.num_threads
Out[12]: 4

In [13]: from copy import deepcopy

In [14]: deepcopy(node._interface).num_threads
Out[14]: 4

will check _make_nodes shortly

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try:

>>> pe.Node(deepcopy(node._interface), name='copynode')._interface.num_threads

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I see the problem: In Node.__init__, self._interface.num_threads = n_procs, which is 1 by default...

Copy link
Member

@satra satra May 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so the issue is with this line:

https://github.com/nipy/nipype/blob/master/nipype/pipeline/engine/nodes.py#L173

therefore in MapNodes, the deepcopy statement should include n_procs=self._interface.num_threads

Node(deepcopy(node._interface), 
     name='copynode', 
     n_procs=node._interface.num_threads)._interface.num_threads

and also any other attribute that's being set there.

@effigies effigies force-pushed the fix/mapnode_parallelism branch from edfcf84 to 6dd54b2 Compare May 14, 2017 13:37
@codecov-io
Copy link

codecov-io commented May 14, 2017

Codecov Report

Merging #2019 into master will increase coverage by <.01%.
The diff coverage is 93.33%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master   #2019      +/-   ##
=========================================
+ Coverage   72.19%   72.2%   +<.01%     
=========================================
  Files        1132    1132              
  Lines       57012   57023      +11     
  Branches     8162    8165       +3     
=========================================
+ Hits        41161   41173      +12     
+ Misses      14567   14566       -1     
  Partials     1284    1284
Flag Coverage Δ
#smoketests 72.2% <93.33%> (ø) ⬆️
#unittests 69.79% <93.33%> (ø) ⬆️
Impacted Files Coverage Δ
nipype/pipeline/engine/nodes.py 84.88% <100%> (+0.21%) ⬆️
nipype/pipeline/engine/tests/test_engine.py 93.52% <92.85%> (-0.03%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4358fe5...28198af. Read the comment docs.

@effigies
Copy link
Member Author

@satra I moved all the parameters that could be moved into the Node(). Seem reasonable?

@effigies effigies force-pushed the fix/mapnode_parallelism branch from 6dd54b2 to f4aab61 Compare May 14, 2017 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants