ResourceMultiProc plugin and runtime profiler

[pintohutch]

1. @carolFrohlich changed the MultiProc plugin to be the ResourceMultiProc plugin. This plugin enables multithreading during workflow execution but also ensures that the workflow will not use more of the system resources than the user specifies with the plugin arguments 'n_procs' and 'memory' at the workflow level. Individual node-level resource usage can be specified via the node._interface.num_threads and node._interface.estimated_memory parameters (defaults are 1 thread and 1 gb, respectively).
2. @carolFrohlich added callback logger that will record estimated node resources and runtimes into a log file via json format.
3. @carolFrohlich added a draw_gantt_chart module which will create a gantt chart graphic html file of the resources utilized by each node during workflow execution from start to finish
4. @dclark87 added a runtime profiler using the psutil and memory_profiler python packages. This profiler will be enabled as long as those packages are installed, and disabled if not installed. The profiler records the actual number of threads and GB of RAM used by a node (as long as it has a nipype interface - will not work with util.function nodes). These values are stored as 'runtime_threads' and 'runtime_memory' in both the runtime Bunch object and result dictionary returned from running the node interface. Because of this, they are also written to the provenance file if that is enabled. The new callback logger will look for these values directly and log them as well.
5. All new features include additional unit tests added to the test folders within each sub-packagae

Thanks to @ccraddock for design and debug help

[chrisgorgo]

KeyError class?

[chrisgorgo]

Ok I finished reading the code. Overall this is a great contribution. Apart from the things I mentioned above the resource_sched_profiler should be listed in https://raw.githubusercontent.com/nipy/nipype/master/doc/users/index.rst

Sorry for picking on the try-catch-all. Being specific about exception classes will pay off in the future. Silently dealing with all potential exceptions makes debugging incredibly hard (believe me, I learned this the hard way!).

[satra]

@dclark87 - one question, is there any reason this information is being kept separate from the provenance record that nipype produces?

[pintohutch]

@satra - there is no reason this information is not in the provenance record. We currently store the runtime profiling info in the node's result.runtime as well as the callback logger, but it doesn't hurt to have it in more places. Where is the provenance code?

[coveralls]

Coverage decreased (-0.6%) to 71.842% when pulling 46f3275 on FCP-INDI:resource_multiproc into aeecd2f on nipy:master.

[coveralls]

Coverage decreased (-0.6%) to 71.842% when pulling 41c6928 on FCP-INDI:resource_multiproc into aeecd2f on nipy:master.

[coveralls]

Coverage decreased (-0.6%) to 71.852% when pulling f0a3889 on FCP-INDI:resource_multiproc into aeecd2f on nipy:master.

[coveralls]

Coverage decreased (-0.5%) to 72.006% when pulling b566b22 on FCP-INDI:resource_multiproc into aeecd2f on nipy:master.

[coveralls]

Coverage decreased (-0.4%) to 72.036% when pulling e7eac16 on FCP-INDI:resource_multiproc into aeecd2f on nipy:master.

[chrisgorgo]

Missing ` ?

[chrisgorgo]

LGTM!