Skip to content

Afni workshop #2135

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Aug 12, 2017
Merged

Afni workshop #2135

merged 19 commits into from
Aug 12, 2017

Conversation

Shotgunosine
Copy link
Collaborator

adding minimal interface wrappers needed to wrap a basic AFNI functional processing pipeline

Copy link
Member

@effigies effigies left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gets less thorough as it goes down, but here is a first pass at a review.

I'll start by saying I'm not super familiar with AFNI, so you should absolutely argue with me if something I suggest doesn't make sense.

@@ -8,7 +8,8 @@
"""

from .base import Info
from .preprocess import (Allineate, Automask, AutoTcorrelate,
from .preprocess import (AlignEpiAnatPy,Allineate, Automask,
AutoTcorrelate,AutoTLRC,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style point: spaces after commas.

#default='/opt/miniconda/envs/py27/bin/python',
mandatory=True,
position=0
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple points:

Shouldn't align_epi_anat.py be an executable in the path with #!/usr/bin/env python27 or similar? If this is really necessary, how about:

py27_path = traits.Either('python2', File(exists=True), usedefault=True, ...)

Further, it might make sense to abstract some of this Python stuff to:

class AFNIPythonCommandInputSpec(AFNICommandInputSpec):
    py27_path = traits.Either('python2', File(exists=True), usedefault=True, ...)

class AFNIPythonCommand(AFNICommand):
    @property
    def cmd(self):
        return spawn.find_executable(super(AFNIPythonCommand, self).cmd)

    @property
    def cmdline(self):
        return "{} {}".format(self.inputs.py27_path, super(AFNIPythonCommand, self).cmdline)

Then you can write the interface consistently with non-Python interfaces.

class AlignEpiAnatPy(AFNIPythonCommand):
    _cmd = 'align_epi_anat.py'
    ....

desc="skull-stripped (not aligned) volume")

class AlignEpiAnatPy(AFNICommand):
"""align EPI to anatomical datasets or vice versa
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Align"

desc='the epi base used in alignment'
'should be one of (0/mean/median/max/subbrick#)',
mandatory=True,
argstr='-epi_base %s')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this is an integer with minimum of 0, or one of the three strings "mean", "median" or "max"?

epi_base = traits.Either(traits.Range(low=0), traits.Enum('mean', 'median', 'max'), ...)

anat2epi = traits.Bool(
default = True,
desc='align anatomical to EPI dataset (default)',
argstr='-anat2epi')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "default = True" here won't have any effect, because you aren't using usedefault=True. The following is probably what you want:

anat2epi = traits.Bool(True, usedefault=True, desc='...', argstr='-anat2epi')

Given that it is the default behavior, it won't make a semantic difference, though...

if not isdefined(self.inputs.suffix):
suffix = '_al'
else:
suffix = '_'+self.inputs.suffix
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the default set as 'al' above, you can replace this if/else with:

suffix = '_' + self.inputs.suffix

else:
suffix = '_'+self.inputs.suffix
if self.inputs.anat2epi:
outputs['anat_al_orig'] = os.path.abspath(self._gen_fname(anat_prefix, suffix=suffix+'+orig')+ext)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFNICommand._gen_fname already prepends the cwd, so you can drop the os.path.abspath. I'm a little confused by the ext here. It looks like, if outputtype == 'NIFTI', the filename will end with _al+orig.nii.HEAD. I'm not very familiar with AFNI's formats, but I'd assume .HEADand.nii` are mutually exclusive.

Perhaps what you want is:

outputtype = self.inputs.outputtype
ext = '.HEAD' if outputtype == 'AFNI' else Info.output_type_to_ext(outputtype)
[...]
outputs['anat_al_orig'] = self._gen_fname(anat_prefix, suffix=suffix + '+orig', ext=ext)

And so-on for the rest.

out_file = File(
desc='collect output to a file',
argstr=' |& tee %s',
position=-1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may be reimplementing terminal_output, though that splits stderr and stdout.

traits.Tuple(
(File(
exists=True,
desc='input file',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think desc makes a difference in internal traits.

@@ -491,6 +701,76 @@ class Copy(AFNICommand):
input_spec = CopyInputSpec
output_spec = AFNICommandOutputSpec

class DotInputSpec(AFNICommandInputSpec):
in_files = traits.List(
(File()),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File(exists=True), presumably?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The check for file existence isn't currently compatible with AFNI's sub-brick selector, until I get around to addressing that, I'd like to leave off the existence requirement for this interface.

@codecov-io
Copy link

codecov-io commented Aug 4, 2017

Codecov Report

Merging #2135 into master will increase coverage by 0.04%.
The diff coverage is 81.01%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2135      +/-   ##
==========================================
+ Coverage   72.19%   72.23%   +0.04%     
==========================================
  Files        1158     1167       +9     
  Lines       58032    58344     +312     
  Branches     8336     8388      +52     
==========================================
+ Hits        41895    42147     +252     
- Misses      14815    14866      +51     
- Partials     1322     1331       +9
Flag Coverage Δ
#smoketests 72.23% <81.01%> (+0.04%) ⬆️
#unittests 69.89% <81.01%> (+0.05%) ⬆️
Impacted Files Coverage Δ
nipype/interfaces/afni/tests/test_auto_Means.py 85.71% <ø> (ø) ⬆️
nipype/interfaces/afni/tests/test_auto_Volreg.py 85.71% <ø> (ø) ⬆️
nipype/interfaces/afni/tests/test_auto_Calc.py 85.71% <ø> (ø) ⬆️
...ipype/interfaces/afni/tests/test_auto_Allineate.py 85.71% <100%> (ø) ⬆️
...terfaces/afni/tests/test_auto_AFNIPythonCommand.py 100% <100%> (ø)
nipype/interfaces/afni/__init__.py 100% <100%> (ø) ⬆️
nipype/interfaces/afni/preprocess.py 81.38% <62.79%> (-2.35%) ⬇️
nipype/interfaces/afni/base.py 54.4% <81.81%> (+2.64%) ⬆️
nipype/interfaces/afni/tests/test_auto_TNorm.py 85.71% <85.71%> (ø)
nipype/interfaces/afni/tests/test_auto_Dot.py 85.71% <85.71%> (ø)
... and 16 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b587d6...1254589. Read the comment docs.

@Shotgunosine
Copy link
Collaborator Author

Shotgunosine commented Aug 7, 2017

hmm, it seems like there's not a reliable place to grab the python 2 executable across all the travis builds. @effigies Do you have any ideas how I could structure the interfaces for the AFNI python scripts to cope with this?

@satra
Copy link
Member

satra commented Aug 7, 2017

@Shotgunosine - perhaps we should do the same thing we did for the nimh workshop and have both environments available.

@Shotgunosine
Copy link
Collaborator Author

@satra would the best way to do that be to modify the .travis.yml?

@satra
Copy link
Member

satra commented Aug 7, 2017

both travis and the docker build process for circle.

@Shotgunosine
Copy link
Collaborator Author

Looks like I misdiagnosed the issue with respect to Travis at least. On the Travis builds without AFNI installed 'distutils.spawn' couldn't find the executable and so it was returning None as the command. It now returns just the command string if it can't find the executable and I ellipsed out the path in the docstring test, so the Travis tests pass.

@Shotgunosine
Copy link
Collaborator Author

Hmm, I'm not sure I understand the CircleCI error. I've copied the Traceback below, does that give anyone any insight into what the issue might be?

Traceback (most recent call last):

  File "/src/nipype/tools/run_examples.py", line 64, in <module>

    run_examples(example, pipelines, data_path, plugin)

  File "/src/nipype/tools/run_examples.py", line 48, in run_examples

    wf.run(plugin=plugin, plugin_args=plugin_args)

  File "/src/nipype/nipype/pipeline/engine/workflows.py", line 596, in run

    write_workflow_prov(execgraph, prov_base, format='all')

  File "/src/nipype/nipype/pipeline/engine/utils.py", line 1291, in write_workflow_prov

    ps.write_provenance(filename, format=format)

  File "/src/nipype/nipype/utils/provenance.py", line 436, in write_provenance

    self.g.serialize(filename + ext, format='rdf', rdf_format=rdf_format)

  File "/usr/local/miniconda/lib/python2.7/site-packages/prov/model.py", line 2402, in serialize

    serializer.serialize(stream, **args)

  File "/usr/local/miniconda/lib/python2.7/site-packages/prov/serializers/provrdf.py", line 97, in serialize

    container.serialize(buf, **newargs)

  File "/usr/local/miniconda/lib/python2.7/site-packages/rdflib/graph.py", line 943, in serialize

    serializer.serialize(stream, base=base, encoding=encoding, **args)

  File "/usr/local/miniconda/lib/python2.7/site-packages/rdflib/plugins/serializers/trig.py", line 57, in serialize

    self.preprocess()

  File "/usr/local/miniconda/lib/python2.7/site-packages/rdflib/plugins/serializers/trig.py", line 35, in preprocess

    self.getQName(context.identifier)

  File "/usr/local/miniconda/lib/python2.7/site-packages/rdflib/plugins/serializers/turtle.py", line 274, in getQName

    prefix = self.addNamespace(prefix, namespace)

  File "/usr/local/miniconda/lib/python2.7/site-packages/rdflib/plugins/serializers/turtle.py", line 202, in addNamespace

    super(TurtleSerializer, self).addNamespace(prefix, namespace)

  File "/usr/local/miniconda/lib/python2.7/site-packages/rdflib/plugins/serializers/turtle.py", line 51, in addNamespace

    raise Exception("Trying to override namespace prefix %s => %s, but it's already bound to %s"%(prefix, uri, self.namespaces[prefix]))

Exception: Trying to override namespace prefix pns1 => http://iri.nidash.org/13, but it's already bound to http://iri.nidash.org/15

+ exitcode=1
+ cp '/home/ubuntu/work/tests/*.xml' /tmp/circle-junit.JmdY77x/tests/
cp: cannot stat ‘/home/ubuntu/work/tests/*.xml’: No such file or directory

@effigies
Copy link
Member

effigies commented Aug 8, 2017

That issue should be resolved by #2137, which occurred after your fork. Merge or rebase against master to get that update.

@Shotgunosine
Copy link
Collaborator Author

Now CircleCI is dying earlier with the following error:

(cd $HOME/docker && gzip -d cache.tar.gz && docker load --input $HOME/docker/cache.tar) || true

(cd $HOME/docker && gzip -d cache.tar.gz && docker load --input $HOME/docker/cache.tar) || true died unexpectedly

Action failed: (cd $HOME/docker && gzip -d cache.tar.gz && docker load --input $HOME/docker/cache.tar) || true

@effigies
Copy link
Member

effigies commented Aug 8, 2017

Restarted the build. Looks like it made it through, this time.

@satra satra merged commit 539fdb9 into nipy:master Aug 12, 2017
@satra satra added this to the 0.14.0 milestone Oct 20, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants