Merge pull request #1177 from chrisfilo/enh/circle_ci

Support for CircleCI

Author: satra

circle.yml

@@ -0,0 +1,51 @@
+dependencies:
+  cache_directories:
+    - "~/examples/data"
+    - "~/examples/fsdata"
+    - "~/examples/feeds"
+    - "~/mcr"
+    - "~/spm12"
+    - "~/examples/fsl_course_data"
+  override:
+    - pip install --upgrade pip
+    - pip install -e .
+    - pip install matplotlib sphinx ipython
+    - if [[ ! -d ~/examples/data ]]; then wget "http://tcpdiag.dl.sourceforge.net/project/nipy/nipype/nipype-0.2/nipype-tutorial.tar.bz2"; tar jxvf nipype-tutorial.tar.bz2; mkdir ~/examples; mv nipype-tutorial/* ~/examples/; fi
+    # we download this manually because CircleCI does not cache apt
+    - if [[ ! -d ~/examples/feeds ]]; then wget "http://fsl.fmrib.ox.ac.uk/fsldownloads/fsl-5.0.8-feeds.tar.gz"; tar zxvf fsl-5.0.8-feeds.tar.gz; mv feeds ~/examples/; fi
+    - if [[ ! -d ~/examples/fsl_course_data ]]; then wget -c "http://fsl.fmrib.ox.ac.uk/fslcourse/fdt1.tar.gz" ; wget -c "http://fsl.fmrib.ox.ac.uk/fslcourse/fdt2.tar.gz"; wget -c "http://fsl.fmrib.ox.ac.uk/fslcourse/tbss.tar.gz"; mkdir ~/examples/fsl_course_data; tar zxvf fdt1.tar.gz -C ~/examples/fsl_course_data; tar zxvf fdt2.tar.gz -C ~/examples/fsl_course_data; tar zxvf tbss.tar.gz -C ~/examples/fsl_course_data; fi
+    - wget -O- http://neuro.debian.net/lists/trusty.us-nh.full | sudo tee /etc/apt/sources.list.d/neurodebian.sources.list
+    - sudo apt-key adv --recv-keys --keyserver hkp://pgp.mit.edu:80 0xA5D32F012649A5A9
+    - sudo apt-get update -y; sudo apt-get install -y fsl-core fsl-atlases
+    - bash ~/nipype/tools/install_spm_mcr.sh
+    - mkdir ~/.nipype && echo "[logging]" > ~/.nipype/nipype.cfg && echo "workflow_level = DEBUG" >> ~/.nipype/nipype.cfg && echo "interface_level = DEBUG" >> ~/.nipype/nipype.cfg && echo "filemanip_level = DEBUG" >> ~/.nipype/nipype.cfg
+test:
+  override:
+    - . /usr/share/fsl/5.0/etc/fslconf/fsl.sh && nosetests --with-doctest --logging-level=DEBUG --verbosity=3:
+        environment:
+          SPMMCRCMD: "$HOME/spm12/run_spm12.sh $HOME/mcr/v85/ script"
+          FORCE_SPMMCR: 1
+          FSL_COURSE_DATA: "$HOME/examples/fsl_course_data"
+        timeout: 2600
+    - set -o pipefail && cd doc && make html 2>&1 | tee ~/log.txt
+    - cat ~/log.txt && if grep -q "ERROR" ~/log.txt; then false; else true; fi
+    - . /usr/share/fsl/5.0/etc/fslconf/fsl.sh && python ~/nipype/tools/run_examples.py fmri_fsl_feeds Linear l1pipeline:
+        pwd: ../examples
+    - . /usr/share/fsl/5.0/etc/fslconf/fsl.sh && python ~/nipype/tools/run_examples.py fmri_spm_dartel Linear level1 l2pipeline:
+        pwd: ../examples
+        environment:
+          SPMMCRCMD: "$HOME/spm12/run_spm12.sh $HOME/mcr/v85/ script"
+          FORCE_SPMMCR: 1
+        timeout: 1600
+    - . /usr/share/fsl/5.0/etc/fslconf/fsl.sh && python ~/nipype/tools/run_examples.py fmri_fsl_reuse Linear level1_workflow:
+        pwd: ../examples
+    - . /usr/share/fsl/5.0/etc/fslconf/fsl.sh && python ~/nipype/tools/run_examples.py fmri_spm_nested Linear level1 l2pipeline:
+        pwd: ../examples
+        environment:
+          SPMMCRCMD: "$HOME/spm12/run_spm12.sh $HOME/mcr/v85/ script"
+          FORCE_SPMMCR: 1
+
+general:
+  artifacts:
+    - "doc/_build/html"
+    - "~/log.txt"

doc/devel/cmd_interface_devel.rst

@@ -21,15 +21,15 @@ grouped into separate classes (usually suffixed with InputSpec and OutputSpec).
 For example:
 
 .. testcode::
-	
+
 	class ExampleInputSpec(TraitedSpec):
 		input_volume = File(desc = "Input volume", exists = True,
 		                    mandatory = True)
 		parameter = traits.Int(desc = "some parameter")
-		
+
 	class ExampleOutputSpec(TraitedSpec):
 		output_volume = File(desc = "Output volume", exists = True)
-		
+
 For the Traits (and Nipype) to work correctly output and input spec has to be
 inherited from TraitedSpec (however, this does not have to be direct
 inheritance).
@@ -39,7 +39,7 @@ above example we have used the ``desc`` metadata which holds human readable
 description of the input. The ``mandatory`` flag forces Nipype to throw an
 exception if the input was not set. ``exists`` is a special flag that works only
 for ``File traits`` and checks if the provided file exists. More details can be
-found at `interface_specs`_.
+found at :doc:`interface_specs`.
 
 The input and output specifications have to be connected to the our example
 interface class:
@@ -49,13 +49,13 @@ interface class:
 	class Example(Interface):
 		input_spec = ExampleInputSpec
 		output_spec = ExampleOutputSpec
-		
+
 Where the names of the classes grouping inputs and outputs were arbitrary the
 names of the fields within the interface they are assigned are not (it always
 has to be input_spec and output_spec). Of course this interface does not do much
 because we have not specified how to process the inputs and create the outputs.
 This can be done in many ways.
- 
+
 Command line executable
 =======================
 
@@ -67,32 +67,32 @@ between the inputs and the generated command line. To achieve this we have
 added two metadata: ``argstr`` (string defining how the argument should be
 formated) and ``position`` (number defining the order of the arguments).
 For example
- 
+
 .. testcode::
 
 	class ExampleInputSpec(CommandLineSpec):
 		input_volume = File(desc = "Input volume", exists = True,
 		                    mandatory = True, position = 0, argstr="%s")
 		parameter = traits.Int(desc = "some parameter", argstr = "--param %d")
-		
+
 As you probably noticed the ``argstr`` is a printf type string with formatting
 symbols. For an input defined in InputSpec to be included into the executed
 commandline ``argstr`` has to be included. Additionally inside the main
 interface class you need to specify the name of the executable by assigning it
 to the ``_cmd`` field. Also the main interface class needs to inherit from
-`CommandLine`_:
+:class:`nipype.interfaces.base.CommandLine`:
 
 .. testcode::
 
 	class Example(CommandLine):
 		_cmd = 'my_command'
 		input_spec = ExampleInputSpec
 		output_spec = ExampleOutputSpec
-		
+
 There is one more thing we need to take care of. When the executable finishes
 processing it will presumably create some output files. We need to know which
 files to look for, check if they exist and expose them to whatever node would
-like to use them. This is done by implementing `_list_outputs`_ method in the
+like to use them. This is done by implementing ``_list_outputs`` method in the
 main interface class. Basically what it does is assigning the expected output
 files to the fields of our output spec:
 
@@ -102,7 +102,7 @@ files to the fields of our output spec:
 		outputs = self.output_spec().get()
 		outputs['output_volume'] = os.path.abspath('name_of_the_file_this_cmd_made.nii')
 		return outputs
-		
+
 Sometimes the inputs need extra parsing before turning into command line
 parameters. For example imagine a parameter selecting between three methods:
 "old", "standard" and "new". Imagine also that the command line accept this as
@@ -122,42 +122,42 @@ numbers. We need to do additional parsing by overloading the following method
 in the main interface class:
 
 .. testcode::
-	
+
 	def _format_arg(self, name, spec, value):
 		if name == 'method':
 		    return spec.argstr%{"old":0, "standard":1, "new":2}[value]
 		return super(Example, self)._format_arg(name, spec, value)
-		
+
 Here is a minimalistic interface for the gzip command:
 
 .. testcode::
-	
+
 	from nipype.interfaces.base import (
-	    TraitedSpec, 
+	    TraitedSpec,
 	    CommandLineInputSpec,
-	    CommandLine, 
+	    CommandLine,
 	    File
 	)
 	import os
-	
+
 	class GZipInputSpec(CommandLineInputSpec):
 	    input_file = File(desc="File", exists=True, mandatory=True, argstr="%s")
-	        
+
 	class GZipOutputSpec(TraitedSpec):
 	    output_file = File(desc = "Zip file", exists = True)
-	        
+
 	class GZipTask(CommandLine):
 	    input_spec = GZipInputSpec
 	    output_spec = GZipOutputSpec
 	    cmd = 'gzip'
-	    
+
 	    def _list_outputs(self):
 	            outputs = self.output_spec().get()
 	            outputs['output_file'] = os.path.abspath(self.inputs.input_file + ".gz")
 	            return outputs
-	            
+
 	if __name__ == '__main__':
-	    
+
 	    zipper = GZipTask(input_file='an_existing_file')
 	    print zipper.cmdline
 	    zipper.run()
@@ -193,9 +193,9 @@ hash_files
 
 name_template (optional)
      overrides the default ``_generated`` suffix
-     
+
 output_name (optional)
-     name of the output (if this is not set same name as the input will be 
+     name of the output (if this is not set same name as the input will be
      assumed)
 
 keep_extension (optional - not used)

doc/index.rst

@@ -4,31 +4,31 @@
          :width: 100 %
    -  .. container::
 
-      Current neuroimaging software offer users an incredible opportunity to
-      analyze data using a variety of different algorithms. However, this has
-      resulted in a heterogeneous collection of specialized applications
-      without transparent interoperability or a uniform operating interface.
-
-      *Nipype*, an open-source, community-developed initiative under the
-      umbrella of NiPy_, is a Python project that provides a uniform interface
-      to existing neuroimaging software and facilitates interaction between
-      these packages within a single workflow. Nipype provides an environment
-      that encourages interactive exploration of algorithms from different
-      packages (e.g., ANTS_, SPM_, FSL_, FreeSurfer_, Camino_, MRtrix_, MNE_, AFNI_, 
-      Slicer_), eases the design of workflows within and between packages, and
-      reduces the learning curve necessary to use different packages. Nipype is
-      creating a collaborative platform for neuroimaging software development 
-      in a high-level language and addressing limitations of existing pipeline
-      systems.
-
-      *Nipype* allows you to:
-
-      * easily interact with tools from different software packages
-      * combine processing steps from different software packages
-      * develop new workflows faster by reusing common steps from old ones
-      * process data faster by running it in parallel on many cores/machines
-      * make your research easily reproducible
-      * share your processing workflows with the community
+        Current neuroimaging software offer users an incredible opportunity to
+        analyze data using a variety of different algorithms. However, this has
+        resulted in a heterogeneous collection of specialized applications
+        without transparent interoperability or a uniform operating interface.
+
+        *Nipype*, an open-source, community-developed initiative under the
+        umbrella of NiPy_, is a Python project that provides a uniform interface
+        to existing neuroimaging software and facilitates interaction between
+        these packages within a single workflow. Nipype provides an environment
+        that encourages interactive exploration of algorithms from different
+        packages (e.g., ANTS_, SPM_, FSL_, FreeSurfer_, Camino_, MRtrix_, MNE_, AFNI_,
+        Slicer_), eases the design of workflows within and between packages, and
+        reduces the learning curve necessary to use different packages. Nipype is
+        creating a collaborative platform for neuroimaging software development
+        in a high-level language and addressing limitations of existing pipeline
+        systems.
+
+        *Nipype* allows you to:
+
+        * easily interact with tools from different software packages
+        * combine processing steps from different software packages
+        * develop new workflows faster by reusing common steps from old ones
+        * process data faster by running it in parallel on many cores/machines
+        * make your research easily reproducible
+        * share your processing workflows with the community
 
 .. admonition:: Reference
 

doc/users/config_file.rst

@@ -80,8 +80,8 @@ Execution
 *remove_unnecessary_outputs*
 	This will remove any interface outputs not needed by the workflow. If the
 	required outputs from a node changes, rerunning the workflow will rerun the
-	node. Outputs of leaf nodes (nodes whose outputs are not connected to any 
-	other nodes) will never be deleted independent of this parameter. (possible 
+	node. Outputs of leaf nodes (nodes whose outputs are not connected to any
+	other nodes) will never be deleted independent of this parameter. (possible
 	values: ``true`` and ``false``; default value: ``true``)
 
 *try_hard_link_datasink*
@@ -129,7 +129,7 @@ Execution
     If this is set to True, the node's output directory will contain full
     parameterization of any iterable, otherwise parameterizations over 32
     characters will be replaced by their hash. (possible values: ``true`` and
-	``false``; default value: ``true``)
+    ``false``; default value: ``true``)
 
 *poll_sleep_duration*
     This controls how long the job submission loop will sleep between submitting
@@ -146,7 +146,7 @@ Example
 
 	[logging]
 	workflow_level = DEBUG
-	
+
 	[execution]
 	stop_on_first_crash = true
 	hash_method = timestamp
@@ -156,9 +156,9 @@ Workflow.config property has a form of a nested dictionary reflecting the
 structure of the .cfg file.
 
 ::
-  
+
   myworkflow = pe.Workflow()
-  myworkflow.config['execution'] = {'stop_on_first_rerun': 'True', 
+  myworkflow.config['execution'] = {'stop_on_first_rerun': 'True',
                                      'hash_method': 'timestamp'}
 
 You can also directly set global config options in your workflow script. An

doc/users/plugins.rst

@@ -71,7 +71,7 @@ a local system.
 
 Optional arguments::
 
-  n_procs :  Number of processes to launch in parallel, if not set number of 
+  n_procs :  Number of processes to launch in parallel, if not set number of
   processors/threads will be automatically detected
 
 To distribute processing on a multicore machine, simply call::
@@ -118,7 +118,7 @@ Optional arguments::
 
 For example, the following snippet executes the workflow on myqueue with
 a custom template::
- 
+
        workflow.run(plugin='SGE',
           plugin_args=dict(template='mytemplate.sh', qsub_args='-q myqueue')
 
@@ -136,18 +136,18 @@ particular node might use more resources than other nodes in a workflow.
   this local configuration::
 
      node.plugin_args = {'qsub_args': '-l nodes=1:ppn=3', 'overwrite': True}
-	 
+
 SGEGraph
 ~~~~~~~~
 SGEGraph_ is an execution plugin working with Sun Grid Engine that allows for
-submitting entire graph of dependent jobs at once. This way Nipype does not 
+submitting entire graph of dependent jobs at once. This way Nipype does not
 need to run a monitoring process - SGE takes care of this. The use of SGEGraph_
-is preferred over SGE_ since the latter adds unnecessary load on the submit 
+is preferred over SGE_ since the latter adds unnecessary load on the submit
 machine.
 
 .. note::
 
-  When rerunning unfinished workflows using SGEGraph you may decide not to 
+  When rerunning unfinished workflows using SGEGraph you may decide not to
   submit jobs for Nodes that previously finished running. This can speed up
   execution, but new or modified inputs that would previously trigger a Node
   to rerun will be ignored. The following option turns on this functionality::
@@ -177,20 +177,20 @@ Optional arguments::
 
   template: custom template file to use
   sbatch_args: any other command line args to be passed to bsub.
-  
-  
+
+
 SLURMGraph
 ~~~~~~~~~~
 SLURMGraph_ is an execution plugin working with SLURM that allows for
-submitting entire graph of dependent jobs at once. This way Nipype does not 
+submitting entire graph of dependent jobs at once. This way Nipype does not
 need to run a monitoring process - SLURM takes care of this. The use of SLURMGraph_
-plugin is preferred over the vanilla SLURM_ plugin since the latter adds 
-unnecessary load on the submit machine. 
+plugin is preferred over the vanilla SLURM_ plugin since the latter adds
+unnecessary load on the submit machine.
 
 
 .. note::
 
-  When rerunning unfinished workflows using SLURMGraph you may decide not to 
+  When rerunning unfinished workflows using SLURMGraph you may decide not to
   submit jobs for Nodes that previously finished running. This can speed up
   execution, but new or modified inputs that would previously trigger a Node
   to rerun will be ignored. The following option turns on this functionality::
@@ -205,11 +205,11 @@ DAGMan
 ~~~~~~
 
 With its DAGMan_ component HTCondor_ (previously Condor) allows for submitting
-entire graphs of dependent jobs at once (similar to SGEGraph_ and SLURMGaaoh_). 
-With the ``CondorDAGMan`` plug-in Nipype can utilize this functionality to 
-submit complete workflows directly and in a single step.  Consequently, and 
-in contrast to other plug-ins, workflow execution returns almost 
-instantaneously -- Nipype is only used to generate the workflow graph, 
+entire graphs of dependent jobs at once (similar to SGEGraph_ and SLURMGraph_). 
+With the ``CondorDAGMan`` plug-in Nipype can utilize this functionality to
+submit complete workflows directly and in a single step.  Consequently, and
+in contrast to other plug-ins, workflow execution returns almost
+instantaneously -- Nipype is only used to generate the workflow graph,
 while job scheduling and dependency resolution are entirely managed by HTCondor_.
 
 Please note that although DAGMan_ supports specification of data dependencies
@@ -320,4 +320,3 @@ Optional arguments::
 .. _HTCondor documentation: http://research.cs.wisc.edu/htcondor/manual
 .. _DMTCP: http://dmtcp.sourceforge.net
 .. _SLURM: http://slurm.schedmd.com/
-