Discuss: Nipype boilerplate simplification/ v2 API

[satra]

Summary

When one writes a nipype interface or workflow there is a fair bit of boilerplate. The intent of this discussion is to figure out how we reduce/simplify this, and to think about the v2 api.

should we:

1. just simplify syntax (allow more restricted forms only)/update our examples to use more compact pythonic forms. for example:

# common form
a.inputs.one = 2
a.inputs.two = 3

# alternate allowed forms
a  = Interface(one=2, two=3)
or
a = Interface(...)
a.inputs.update(one=2, two=3)

except for special interfaces (e.g., Function)

2. add support for lazy evaluation in new api and make it more compact (my current preference):

wf2=Workflow('wf12, ..., distribute=False, global_cache=False)
...
wf1=Workflow('wf1', ..., distribute=True)
wf1.add('a', BET(...))
wf1.add('wf2', wf2(..., inputs='node2.in1=a.out1'))
wf1.add('b', antsRegistration(..., in1='a.out1')).map('in1'))
wf1.add('x', SomeFunc(..., in1='b.out1')) # map automatically continues
wf1.add('c', ImageMaths(..., in1='wf2.node1.out1')).join('b.in1')
wf1.run()

or more compactly:

wf2=Workflow('wf2', ..., distribute=False, global_cache=False)
...
wf1=Workflow('wf1', ..., distribute=True)
    .add('a', BET(...))
    .add('wf2', wf2(..., inputs='node2.in1=a.out1'))
    .add('b', antsRegistration(..., in1='a.out1')).map('in1'))
    .add('x', SomeFunc(..., in1='b.out1')) # map automatically continues
    .add('c', ImageMaths(..., in1='wf2.node1.out1')).join('b.in1')
wf1.run()

.add will return a composite reference to both the workflow and the interface added

3. create a DSL (domain specific language - like WDL/CWL) instead of using Python

@djarecka @Shotgunosine @effigies @chrisfilo @oesteban - open for discussion.

[satra]

instead of 3, we should also be able to do:

wf1 = Workflow(...).from_cwl('/path/to/cwl')

would use the cwl library to parse the file and convert to a nipype workflow.

and correspondingly wf1.to_cwl()/boutiques()

[satra]

also for map and join, i'm thinking we could do something like:

1. map
   .add('b', antsRegistration(..., in1='a.out1')).map('in1', propagate_failures=True))

True can be default with False, failure_callback being options.

2. join

.add('c', ImageMaths(..., in1='wf2.node1.out1')).join('b.in1', failure_mode='allow_any')

options could be allow_any | allow_none, we could mark failed outputs as failed or xfail or nan or None, and for map processing we can allow failures to propagate

[effigies]

How about this as a Python-based DSL?

class Workflow1(Workflow):
    a = BET(...)
    wf2 = Workflow2(..., inputs={'node2.in1': a.out1})
    b = Map(antsRegistration(..., in1=a.out1), 'in1', propagate_failures=True)
    x = SomeFunc(..., in1=b.out1)
    c = Join(ImageMaths(..., in1=wf2.node1.out1), 'b.in1', failure_mode='allow_any')

wf1 = Workflow1()
wf1.run()

This seems like it would have to be something of syntactic sugar, and __init__ would need to do the work of turning these class variables into independent instance variables. I'm not sure if there's something in your proposed syntax that would be very difficult to translate into this approach.

[satra]

@effigies - i can live with that and construct a graph from this.

[oesteban]

I feel that @effigies' suggestion would also allow to easily set up conditional nodes and other decision making within the definition of the Workflow

[satra]

@oesteban - indeed I was about to bring up conditionals and loops. I'm not sure how that would fit in to the dsl yet. We should flesh that piece out before settling on the API.

[Shotgunosine]

@effigies How much overhead is that going to add to interface creation?

[effigies]

@satra I would be interested in conditionals and loops in the format suggested in (2), as well. And MapNode, as well. Would that just be .map(...).join(...), and thus Join(Map(...), ...) in mine?

Also, it would be good to translate a real workflow - that uses iterables, JoinNode and MapNode - into each syntax. If I can suggest one from fmriprep: init_bold_surf_wf.

@Shotgunosine Overhead in what sense?

[Shotgunosine]

I mean when you say that init would need to do the work of turning the class variables into instance variables. Is that work that people adding interfaces need to do?

[leej3]

My two cents: I really like the compact @satra outlines. I'm a big fan of functional programming so it seems very natural to me. I think that would be a pleasing format to teach to a complete beginner too though.

[Shotgunosine]

I do like the compact form of 2 above as well.

[effigies]

Oh, I see. No, I meant that we would need to code the base Workflow class with an __init__ that makes a usable workflow from the declarative style above.

That said, often workflow setup does need to be dynamically adjusted at runtime, which could be handled like so:

class MyWorkflow(Workflow):
    ...
    def __init__(self, ...):
        super(MyWorkflow, self).__init__(self, ...)
        # Workflow is fully initialized, and any runtime
        # additions/modifications can be done here

We could also approach my syntax as sugar on top of form 2, so post-initializing setup could be done outside the workflow itself:

wf = MyWorkflow(distribute=True)
if inputs_look_like_this:
    wf.add('y', SomeOtherNode(...))
else:
    wf.add('z', YetAnother(...))

To be clear, I think form 2 looks reasonable, and I only arrived at my approach by looking at it and thinking what could be cut further. This replaces (perhaps optionally?) wf.add('b', node) with wf.b = node, which will naturally result in more work behind the scenes. (Hopefully not too much.)