OSError on old_cwd for IPython Plugin

[blakedewey]

I have seen this a few places around, but there was never an issue placed. When running with the IPython plugin, I am getting random errors where os.getcwd() fails for a node execution. I can't find any explanation. I am including the pypeline.log and the terminal output. Let me know what you guys think. The error is at line 176 in the terminal output.

[blakedewey]

pypeline.log:

141030-16:54:09,951 workflow INFO:
     Executing node casedata.aI.a0 in dir: /tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/casedata
141030-16:54:09,952 workflow DEBUG:
     []
141030-16:54:09,952 workflow DEBUG:
     /tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/casedata/_0x27fbf7891c44d5767d3b0655464004ec.json
141030-16:54:09,952 workflow DEBUG:
     ('updatehash, overwrite, always_run, hash_exists', False, None, False, False)
141030-16:54:09,952 workflow DEBUG:
     Node hash: 27fbf7891c44d5767d3b0655464004ec
141030-16:54:09,952 workflow DEBUG:
     /tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/casedata/_0x27fbf7891c44d5767d3b0655464004ec_unfinished.json found and can_resume is True or Node is a MapNode - resuming execution
141030-16:54:09,952 workflow DEBUG:
     Creating /tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/casedata
141030-16:54:09,953 workflow DEBUG:
     writing pre-exec report to /tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/casedata/_report/report.rst
141030-16:54:09,955 workflow DEBUG:
     Executing node
141030-16:54:09,956 workflow DEBUG:
     copying files to wd [execute=True, linksonly=False]
141030-16:54:09,961 workflow DEBUG:
     Needed files: /tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/casedata/_0x27fbf7891c44d5767d3b0655464004ec_unfinished.json;/tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/casedata/_inputs.pklz;/tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/casedata/_node.pklz
141030-16:54:09,962 workflow DEBUG:
     Needed dirs: /tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/casedata/_report
141030-16:54:09,962 workflow DEBUG:
     Removing files: 
141030-16:54:09,963 workflow DEBUG:
     saved results in /tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/casedata/result_casedata.pklz
141030-16:54:09,964 workflow DEBUG:
     writing post-exec report to /tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/casedata/_report/report.rst
141030-16:54:09,964 workflow DEBUG:
     Finished running casedata.aI.a0 in dir: /tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/casedata

141030-16:54:11,966 workflow INFO:
     Executing node inputdata.a0 in dir: /tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/inputdata
141030-16:54:11,967 workflow DEBUG:
     []
141030-16:54:11,967 workflow DEBUG:
     /tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/inputdata/_0xcb2c6c0905d5295dfd7a7637f5c413b8.json
141030-16:54:11,967 workflow DEBUG:
     ('updatehash, overwrite, always_run, hash_exists', False, None, True, False)
141030-16:54:11,968 workflow DEBUG:
     Node hash: cb2c6c0905d5295dfd7a7637f5c413b8
141030-16:54:11,968 workflow DEBUG:
     /tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/inputdata/_0xcb2c6c0905d5295dfd7a7637f5c413b8_unfinished.json found and can_resume is True or Node is a MapNode - resuming execution
141030-16:54:11,968 workflow DEBUG:
     Creating /tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/inputdata
141030-16:54:11,969 workflow DEBUG:
     writing pre-exec report to /tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/inputdata/_report/report.rst
141030-16:54:11,972 workflow DEBUG:
     Aggregate: True

[blakedewey]

terminal output:

141030-16:54:09,890 workflow DEBUG:
         adding multipath trait: config_file
141030-16:54:09,892 workflow DEBUG:         
         adding multipath trait: echos      
141030-16:54:09,893 workflow DEBUG:         
         adding multipath trait: echo_configs
141030-16:54:09,893 workflow DEBUG:          
         adding multipath trait: tag         
141030-16:54:09,894 workflow DEBUG:          
         adding multipath trait: in_file     
141030-16:54:09,894 workflow DEBUG:          
         adding multipath trait: output_name 
141030-16:54:09,895 workflow DEBUG:          
         adding multipath trait: in_file     
141030-16:54:09,896 workflow DEBUG:          
         adding multipath trait: output_name 
141030-16:54:09,901 workflow DEBUG:          
         (memprage.casedata, memprage.inputdata): No edge data
141030-16:54:09,901 workflow DEBUG:                           
         (memprage.casedata, memprage.inputdata): new edge data: {'connect': [('case', 'case')]}                                                                      
141030-16:54:09,901 workflow DEBUG:                                                
         (memprage.inputdata, memprage.get_config_info): No edge data              
141030-16:54:09,901 workflow DEBUG:                                                
         (memprage.inputdata, memprage.get_config_info): new edge data: {'connect': [('configs', 'config_file')]}                                                     
141030-16:54:09,901 workflow DEBUG:                                                
         (memprage.get_config_info, memprage.sort_memprage): No edge data          
141030-16:54:09,902 workflow DEBUG:                                                
         (memprage.get_config_info, memprage.sort_memprage): new edge data: {'connect': [('series_tag', 'tags')]}                                                     
141030-16:54:09,902 workflow DEBUG:                                                
         (memprage.inputdata, memprage.sort_memprage): No edge data                
141030-16:54:09,902 workflow DEBUG:                                                
         (memprage.inputdata, memprage.sort_memprage): new edge data: {'connect': [('images', 'images'), ('configs', 'configs')]}                                     
141030-16:54:09,902 workflow DEBUG:                                                
         (memprage.sort_memprage, memprage.get_memprage_config_file): No edge data 
141030-16:54:09,902 workflow DEBUG:                                                
         (memprage.sort_memprage, memprage.get_memprage_config_file): new edge data: {'connect': [('sorted_configs', 'echo_configs'), ('unique_tags', 'tag')]}        
141030-16:54:09,902 workflow DEBUG:                                                
         (memprage.sort_memprage, memprage.combine_memprage): No edge data         
141030-16:54:09,902 workflow DEBUG:                                                
         (memprage.sort_memprage, memprage.combine_memprage): new edge data: {'connect': [('sorted_images', 'echos')]}                                                
141030-16:54:09,902 workflow DEBUG:                                                
         (memprage.combine_memprage, memprage.move_memprage): No edge data         
141030-16:54:09,903 workflow DEBUG:                                                
         (memprage.combine_memprage, memprage.move_memprage): new edge data: {'connect': [('combined_image', 'in_file')]}                                             
141030-16:54:09,903 workflow DEBUG:                                                
         (memprage.sort_memprage, memprage.move_memprage): No edge data            
141030-16:54:09,903 workflow DEBUG:                                                
         (memprage.sort_memprage, memprage.move_memprage): new edge data: {'connect': [('unique_tags', 'output_name')]}                                               
141030-16:54:09,903 workflow DEBUG:                                                
         (memprage.casedata, memprage.move_memprage): No edge data                 
141030-16:54:09,903 workflow DEBUG:                                                
         (memprage.casedata, memprage.move_memprage): new edge data: {'connect': [('case', 'case')]}                                                                  
141030-16:54:09,903 workflow DEBUG:                                                
         (memprage.get_memprage_config_file, memprage.move_config): No edge data   
141030-16:54:09,903 workflow DEBUG:                                                
         (memprage.get_memprage_config_file, memprage.move_config): new edge data: {'connect': [('config_file', 'in_file')]}                                          
141030-16:54:09,904 workflow DEBUG:                                                
         (memprage.sort_memprage, memprage.move_config): No edge data              
141030-16:54:09,904 workflow DEBUG:                                                
         (memprage.sort_memprage, memprage.move_config): new edge data: {'connect': [('unique_tags', 'output_name')]}                                                 
141030-16:54:09,904 workflow DEBUG:                                                
         (memprage.casedata, memprage.move_config): No edge data                   
141030-16:54:09,904 workflow DEBUG:                                                
         (memprage.casedata, memprage.move_config): new edge data: {'connect': [('case', 'case')]}                                                                    
141030-16:54:09,904 workflow DEBUG:                                                
         (memprage.move_memprage, memprage.success_node): No edge data             
141030-16:54:09,904 workflow DEBUG:                                                
         (memprage.move_memprage, memprage.success_node): new edge data: {'connect': [('out_file', 'move_memprage')]}                                                 
141030-16:54:09,904 workflow DEBUG:                                                
         (memprage.move_config, memprage.success_node): No edge data               
141030-16:54:09,904 workflow DEBUG:                                                
         (memprage.move_config, memprage.success_node): new edge data: {'connect': [('out_file', 'move_config')]}                                                     
141030-16:54:09,905 workflow DEBUG:                                                
         (memprage.casedata, memprage.success_node): No edge data                  
141030-16:54:09,905 workflow DEBUG:                                                
         (memprage.casedata, memprage.success_node): new edge data: {'connect': [('case', 'passthrough')]}                                                            
141030-16:54:09,905 workflow DEBUG:                                                
         Creating flat graph for workflow: memprage                                
141030-16:54:09,909 workflow DEBUG:                                                
         expanding workflow: memprage                                              
141030-16:54:09,910 workflow DEBUG:                                                
         processing node: memprage.casedata                                        
141030-16:54:09,910 workflow DEBUG:                                                
         processing node: memprage.inputdata                                       
141030-16:54:09,910 workflow DEBUG:                                                
         processing node: memprage.get_config_info                                 
141030-16:54:09,910 workflow DEBUG:                                                
         processing node: memprage.sort_memprage                                   
141030-16:54:09,910 workflow DEBUG:                                                
         processing node: memprage.combine_memprage                                
141030-16:54:09,910 workflow DEBUG:                                                
         processing node: memprage.move_memprage                                   
141030-16:54:09,910 workflow DEBUG:                                                
         processing node: memprage.get_memprage_config_file                        
141030-16:54:09,910 workflow DEBUG:                                                
         processing node: memprage.move_config                                     
141030-16:54:09,911 workflow DEBUG:                                                
         processing node: memprage.success_node                                    
141030-16:54:09,911 workflow DEBUG:                                                
         finished expanding workflow: memprage                                     
141030-16:54:09,911 workflow INFO:                                                 
         ['check', 'execution', 'logging']                                         
141030-16:54:09,915 workflow DEBUG:                                                
         PE: expanding iterables                                                   
141030-16:54:09,915 workflow DEBUG:                                                
         Detected iterable nodes [memprage.casedata]                               
141030-16:54:09,915 workflow DEBUG:                                                
         Expanding the iterable node memprage.casedata...                          
141030-16:54:09,916 workflow DEBUG:                                                
         node: memprage.casedata iterables: {'case': <function <lambda> at 0x57f6230>}                                                                                
141030-16:54:09,916 workflow DEBUG:                                                
         ('subnodes:', [memprage.casedata, memprage.success_node, memprage.move_memprage, memprage.inputdata, memprage.get_config_info, memprage.sort_memprage, memprage.move_config, memprage.combine_memprage, memprage.get_memprage_config_file])     
141030-16:54:09,920 workflow DEBUG:                                                
         setting nodelevel(memprage.casedata.aI) input case = Patient1/20140202-4000215250                                                                            
141030-16:54:09,920 workflow DEBUG:                                                
         PE: expanding iterables ... done                                          
141030-16:54:09,925 workflow DEBUG:                                                
         Performing depth first search                                             
141030-16:54:09,939 workflow INFO:                                                 
         Running in parallel.                                                      
141030-16:54:09,940 workflow DEBUG:                                                
         Number of pending tasks: 0                                                
141030-16:54:09,940 workflow DEBUG:                                                
         Slots available: None                                                     
141030-16:54:09,941 workflow INFO:                                                 
         Submitting 1 jobs                                                         
141030-16:54:09,941 workflow INFO:                                                 
         Executing: casedata.aI.a0 ID: 0                                           
141030-16:54:09,941 workflow DEBUG:                                                
         checking hash locally                                                     
141030-16:54:09,942 workflow DEBUG:                                                
         Setting node inputs                                                       
141030-16:54:09,942 workflow DEBUG:                                                
         []                                                                        
141030-16:54:09,942 workflow DEBUG:                                                
         /tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/casedata/_0x27fbf7891c44d5767d3b0655464004ec.json                                     
141030-16:54:09,942 workflow DEBUG:                                                
         Hash exists False                                                         
141030-16:54:09,942 workflow DEBUG:                                                
         Finished checking hash True                                               
141030-16:54:11,949 workflow INFO:                                                 
         [Job finished] jobname: casedata.aI.a0 jobid: 0                           
141030-16:54:11,951 workflow DEBUG:                                                
         Clearing id: 1                                                            
141030-16:54:11,953 workflow DEBUG:                                                
         Number of pending tasks: 0                                                
141030-16:54:11,953 workflow DEBUG:                                                
         Slots available: None                                                     
141030-16:54:11,954 workflow INFO:                                                 
         Submitting 1 jobs                                                         
141030-16:54:11,954 workflow INFO:                                                 
         Executing: inputdata.a0 ID: 1                                             
141030-16:54:11,954 workflow DEBUG:                                                
         checking hash locally                                                     
141030-16:54:11,955 workflow DEBUG:                                                
         Setting node inputs                                                       
141030-16:54:11,955 workflow DEBUG:                                                
         input: case                                                               
141030-16:54:11,955 workflow DEBUG:                                                
         results file: /tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/casedata/result_casedata.pklz                                           
141030-16:54:11,958 workflow DEBUG:                                                
         output: case                                                              
141030-16:54:11,958 workflow DEBUG:                                                
         setting nodelevel(memprage.inputdata.a0) input case = Patient1/20140202-4000215250                                                                           
141030-16:54:11,959 workflow DEBUG:                                                
         []                                                                        
141030-16:54:11,959 workflow DEBUG:                                                
         /tmp/deweybe/PipelineTest/memprage/_case_Patient1..20140202-4000215250/inputdata/_0xcb2c6c0905d5295dfd7a7637f5c413b8.json                                    
141030-16:54:11,960 workflow DEBUG:                                                
         Hash exists False                                                         
141030-16:54:11,960 workflow DEBUG:                                                
         Finished checking hash True                                               
141030-16:54:13,966 workflow DEBUG:                                                
         Aggregate: True                                                           
141030-16:54:13,966 workflow ERROR:                                                
         ['Node inputdata.a0 failed to run on host jokhang.ninds.nih.gov.']        
141030-16:54:13,966 workflow INFO:                                                 
         Saving crash info to /tmp/deweybe/PipelineTest/crash-20141030-165413-deweybe-inputdata.a0.pklz                                                               
141030-16:54:13,966 workflow INFO:                                                 
         Traceback (most recent call last):                                        
  File "/p4/PipelineSoftware/python/lib/python2.7/site-packages/nipype/pipeline/plugins/ipython.py", line 30, in execute_task                                         
    result = task.run(updatehash=updatehash)                                       
  File "/p4/PipelineSoftware/python/lib/python2.7/site-packages/nipype/pipeline/engine.py", line 1424, in run                                                         
    self._run_interface()                                                          
  File "/p4/PipelineSoftware/python/lib/python2.7/site-packages/nipype/pipeline/engine.py", line 1532, in _run_interface                                              
    old_cwd = os.getcwd()                                                          
OSError: [Errno 2] No such file or directory                                       

141030-16:54:13,968 workflow DEBUG:
         Clearing id: 2            
141030-16:54:13,970 workflow DEBUG:
         Number of pending tasks: 0
141030-16:54:13,970 workflow DEBUG:
         Slots available: None     
141030-16:54:15,972 workflow INFO: 
         ***********************************
141030-16:54:15,972 workflow ERROR:         
         could not run node: memprage.inputdata.a0
141030-16:54:15,973 workflow INFO:                
         crashfile: /tmp/deweybe/PipelineTest/crash-20141030-165413-deweybe-inputdata.a0.pklz                                                                         
141030-16:54:15,973 workflow DEBUG:                                                
         The following dependent nodes were not run                                
141030-16:54:15,973 workflow DEBUG:                                                
         inputdata.a0                                                              
141030-16:54:15,973 workflow DEBUG:                                                
         get_config_info.a0                                                        
141030-16:54:15,973 workflow DEBUG:                                                
         sort_memprage.a0                                                          
141030-16:54:15,973 workflow DEBUG:                                                
         move_config.a0                                                            
141030-16:54:15,973 workflow DEBUG:                                                
         success_node.a0                                                           
141030-16:54:15,973 workflow DEBUG:                                                
         move_memprage.a0                                                          
141030-16:54:15,973 workflow DEBUG:                                                
         get_memprage_config_file.a0                                               
141030-16:54:15,974 workflow DEBUG:                                                
         combine_memprage.a0                                                       
141030-16:54:15,974 workflow INFO:

[satra]

@blakedewey - a few questions:

1. which versions of nipype and ipython
2. which ipython cluster configuration (ssh, pbs, sge ?)
3. is the filesystem consistent across all the machines.

[blakedewey]

@satra - a few answers:

1. nipype: 0.10.0, ipython: 2.3.0
2. IPython cluster is local. As a proof of concept, I am using it on one machine as a task management system.
3. Filesystem is shared between all machines, but as I said I am only using one machine as a proof of concept.

[satra]

now that error is quite weird. if you delete the working directory and run it with MultiProc, do you see the same error?

[blakedewey]

Running with MultiProc does not give this error. I would use some other task management, but right now IPython is the simplest setup. I am looking for more fully functional alternatives without a huge IT investment.

[miykael]

I can confirm the same issue with IPython, when used in parallel on a local cluster. (nipype: 0.10.0, ipython: 2.3.0). If I run the same code with MultiProc it works, with the IPython Plugin it crashes with the following error message (same as @blakedewey)

ERROR:workflow:['Node selectfiles.a0 failed to run on host mnotter.']
INFO:workflow:Saving crash info to ~/nipype_tutorial/crash-...-selectfiles.a0.pklz
INFO:workflow:Traceback (most recent call last):
  File "anaconda/lib/python2.7/site-packages/nipype/pipeline/plugins/ipython.py",
        line 30, in execute_task result = task.run(updatehash=updatehash)
  File "anaconda/lib/python2.7/site-packages/nipype/pipeline/engine.py",
        line 1424, in run self._run_interface()
  File "anaconda/lib/python2.7/site-packages/nipype/pipeline/engine.py",
        line 1532, in _run_interface old_cwd = os.getcwd()
OSError: [Errno 2] No such file or directory

In my case, this error only occurs with an io.SelectFiles or utility.Function node, if it occurs. Sometimes, I'm unable to recreate the error, other times I'm not able to get rid of it. @blakedewey are you observing similar behaviour?

[blakedewey]

@miykael, yes I have seen the same results. Mine reproduces fairly randomly on a run because I have so many Function nodes to do data manipulation. Also, I think I have rarely seen it act up in other nodes, but I can't remember off the top of my head. @chrisfilo, I think you answered a question about this back in 2011 in the nipy-devel mailing list (http://mail.scipy.org/pipermail/nipy-devel/2011-March/005992.html). Do you have any ideas? I tried the solution in that post, but clearing the working directory and restarting the cluster did not seem to help anymore.

[satra]

@blakedewey and @miykael - what platform are you running on?

i looked at the code and if this is failing this has something to do with ipython's load_balanced_view starting in a temp directory or something that is deleted. or starting without a shell. i don't understand why this would be intermittent, unless the temp directory is full. the relevant pieces of nipype code are fairly trivial in this context.

i'm also assuming that you are running with workflow.base_dir set to a non-temp location.

[blakedewey]

I actually do set base_dir to /tmp/my_username to use as scratch space. But I found a similar error when I set the base_dir to a NFS mount.

[miykael]

I'm running on Ubuntu 14.04.1 LTS. I set base_dir of the main workflow to a folder under my user profile. What I don't do is specifying the base_dir of any sub-workflows (preprocessing, 1st-level analysis, etc). The nodes which crash are located in the main workflow, but are connect to nodes in the sub-workflows.

[blakedewey]

I'm on CentOS 6.5. I also have some subworkflows but I see this happening across the board, even when I do only one workflow at a time.

[satra]

could you try running this simple test on your computers and see if it breaks? reset the base_dir to something else.

def get_dir(n):
    import os
    from time import sleep
    cwd = os.getcwd()
    sleep(2)
    return cwd

from nipype import Node, Function, Workflow

wf = Workflow('test_ipy')
wf.base_dir = '/software/temp/nipype-tutorial/temp/'
for i in range(30):
    node  = Node(Function(input_names=['n'], output_names=['cwd'], function=get_dir), 
                 name='get_dir_%02d' % i)
    node.inputs.n = i
    wf.add_nodes([node])

eg = wf.run('IPython')
print('\n'.join([node.result.outputs.cwd for node in eg.nodes()]))

[blakedewey]

Ran 10 times without error.

141031-12:28:56,163 workflow DEBUG:
     Creating flat graph for workflow: test_ipy
141031-12:28:56,171 workflow DEBUG:
     expanding workflow: test_ipy
141031-12:28:56,171 workflow DEBUG:
     processing node: test_ipy.get_dir_15
141031-12:28:56,171 workflow DEBUG:
     processing node: test_ipy.get_dir_10
141031-12:28:56,171 workflow DEBUG:
     processing node: test_ipy.get_dir_27
141031-12:28:56,172 workflow DEBUG:
     processing node: test_ipy.get_dir_00
141031-12:28:56,172 workflow DEBUG:
     processing node: test_ipy.get_dir_29
141031-12:28:56,172 workflow DEBUG:
     processing node: test_ipy.get_dir_01
141031-12:28:56,172 workflow DEBUG:
     processing node: test_ipy.get_dir_02
141031-12:28:56,172 workflow DEBUG:
     processing node: test_ipy.get_dir_24
141031-12:28:56,172 workflow DEBUG:
     processing node: test_ipy.get_dir_04
141031-12:28:56,172 workflow DEBUG:
     processing node: test_ipy.get_dir_18
141031-12:28:56,172 workflow DEBUG:
     processing node: test_ipy.get_dir_09
141031-12:28:56,172 workflow DEBUG:
     processing node: test_ipy.get_dir_23
141031-12:28:56,173 workflow DEBUG:
     processing node: test_ipy.get_dir_06
141031-12:28:56,173 workflow DEBUG:
     processing node: test_ipy.get_dir_25
141031-12:28:56,173 workflow DEBUG:
     processing node: test_ipy.get_dir_21
141031-12:28:56,173 workflow DEBUG:
     processing node: test_ipy.get_dir_20
141031-12:28:56,173 workflow DEBUG:
     processing node: test_ipy.get_dir_26
141031-12:28:56,173 workflow DEBUG:
     processing node: test_ipy.get_dir_13
141031-12:28:56,173 workflow DEBUG:
     processing node: test_ipy.get_dir_17
141031-12:28:56,173 workflow DEBUG:
     processing node: test_ipy.get_dir_08
141031-12:28:56,174 workflow DEBUG:
     processing node: test_ipy.get_dir_07
141031-12:28:56,174 workflow DEBUG:
     processing node: test_ipy.get_dir_05
141031-12:28:56,174 workflow DEBUG:
     processing node: test_ipy.get_dir_19
141031-12:28:56,174 workflow DEBUG:
     processing node: test_ipy.get_dir_03
141031-12:28:56,174 workflow DEBUG:
     processing node: test_ipy.get_dir_12
141031-12:28:56,174 workflow DEBUG:
     processing node: test_ipy.get_dir_22
141031-12:28:56,174 workflow DEBUG:
     processing node: test_ipy.get_dir_14
141031-12:28:56,174 workflow DEBUG:
     processing node: test_ipy.get_dir_28
141031-12:28:56,175 workflow DEBUG:
     processing node: test_ipy.get_dir_11
141031-12:28:56,175 workflow DEBUG:
     processing node: test_ipy.get_dir_16
141031-12:28:56,175 workflow DEBUG:
     finished expanding workflow: test_ipy
141031-12:28:56,175 workflow INFO:
     ['check', 'execution', 'logging']
141031-12:28:56,183 workflow DEBUG:
     PE: expanding iterables
141031-12:28:56,183 workflow DEBUG:
     Detected iterable nodes []
141031-12:28:56,183 workflow DEBUG:
     PE: expanding iterables ... done
141031-12:28:56,188 workflow DEBUG:
     Performing depth first search
141031-12:28:56,201 workflow INFO:
     Running in parallel.
141031-12:28:56,202 workflow DEBUG:
     Number of pending tasks: 0
141031-12:28:56,202 workflow DEBUG:
     Slots available: None
141031-12:28:56,203 workflow INFO:
     Submitting 30 jobs
141031-12:28:56,203 workflow INFO:
     Executing: get_dir_25 ID: 0
141031-12:28:56,203 workflow DEBUG:
     checking hash locally
141031-12:28:56,203 workflow DEBUG:
     Setting node inputs
141031-12:28:56,204 workflow DEBUG:
     []
141031-12:28:56,204 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_25/_0x4a1e5b3e4c74ce2126eeaf69dc93e1ba.json
141031-12:28:56,204 workflow DEBUG:
     Hash exists False
141031-12:28:56,204 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,206 workflow INFO:
     Executing: get_dir_00 ID: 1
141031-12:28:56,206 workflow DEBUG:
     checking hash locally
141031-12:28:56,206 workflow DEBUG:
     Setting node inputs
141031-12:28:56,206 workflow DEBUG:
     []
141031-12:28:56,207 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_00/_0xf3deebcce34c46318bc1cec57f5a68b1.json
141031-12:28:56,207 workflow DEBUG:
     Hash exists False
141031-12:28:56,207 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,208 workflow INFO:
     Executing: get_dir_05 ID: 2
141031-12:28:56,209 workflow DEBUG:
     checking hash locally
141031-12:28:56,209 workflow DEBUG:
     Setting node inputs
141031-12:28:56,209 workflow DEBUG:
     []
141031-12:28:56,210 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_05/_0xedd0b7a8ae5a5e9eaf6edc706642a778.json
141031-12:28:56,210 workflow DEBUG:
     Hash exists False
141031-12:28:56,210 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,212 workflow INFO:
     Executing: get_dir_06 ID: 3
141031-12:28:56,212 workflow DEBUG:
     checking hash locally
141031-12:28:56,212 workflow DEBUG:
     Setting node inputs
141031-12:28:56,212 workflow DEBUG:
     []
141031-12:28:56,212 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_06/_0x2644e3f30997550808f1c861b9442a2e.json
141031-12:28:56,213 workflow DEBUG:
     Hash exists False
141031-12:28:56,213 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,214 workflow INFO:
     Executing: get_dir_23 ID: 4
141031-12:28:56,214 workflow DEBUG:
     checking hash locally
141031-12:28:56,215 workflow DEBUG:
     Setting node inputs
141031-12:28:56,215 workflow DEBUG:
     []
141031-12:28:56,215 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_23/_0x8c8201cf328b48bd87c5d5bf976e346e.json
141031-12:28:56,215 workflow DEBUG:
     Hash exists False
141031-12:28:56,215 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,217 workflow INFO:
     Executing: get_dir_15 ID: 5
141031-12:28:56,217 workflow DEBUG:
     checking hash locally
141031-12:28:56,217 workflow DEBUG:
     Setting node inputs
141031-12:28:56,218 workflow DEBUG:
     []
141031-12:28:56,218 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_15/_0xce3ed5b3e4622f3a6269381ff6ca5fc1.json
141031-12:28:56,218 workflow DEBUG:
     Hash exists False
141031-12:28:56,218 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,220 workflow INFO:
     Executing: get_dir_21 ID: 6
141031-12:28:56,220 workflow DEBUG:
     checking hash locally
141031-12:28:56,220 workflow DEBUG:
     Setting node inputs
141031-12:28:56,221 workflow DEBUG:
     []
141031-12:28:56,221 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_21/_0x829723e2d37facd2adcd7641958c3b69.json
141031-12:28:56,221 workflow DEBUG:
     Hash exists False
141031-12:28:56,221 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,223 workflow INFO:
     Executing: get_dir_09 ID: 7
141031-12:28:56,223 workflow DEBUG:
     checking hash locally
141031-12:28:56,223 workflow DEBUG:
     Setting node inputs
141031-12:28:56,223 workflow DEBUG:
     []
141031-12:28:56,223 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_09/_0x7f9e5ba1d69199773c7938af10519cb6.json
141031-12:28:56,224 workflow DEBUG:
     Hash exists False
141031-12:28:56,224 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,225 workflow INFO:
     Executing: get_dir_11 ID: 8
141031-12:28:56,226 workflow DEBUG:
     checking hash locally
141031-12:28:56,226 workflow DEBUG:
     Setting node inputs
141031-12:28:56,226 workflow DEBUG:
     []
141031-12:28:56,226 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_11/_0xcaf46f09c4b928d00b0085524aa1d698.json
141031-12:28:56,226 workflow DEBUG:
     Hash exists False
141031-12:28:56,227 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,228 workflow INFO:
     Executing: get_dir_13 ID: 9
141031-12:28:56,228 workflow DEBUG:
     checking hash locally
141031-12:28:56,229 workflow DEBUG:
     Setting node inputs
141031-12:28:56,229 workflow DEBUG:
     []
141031-12:28:56,229 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_13/_0x4e71c5be18aa3112e56722adb1b507b3.json
141031-12:28:56,230 workflow DEBUG:
     Hash exists False
141031-12:28:56,230 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,231 workflow INFO:
     Executing: get_dir_03 ID: 10
141031-12:28:56,231 workflow DEBUG:
     checking hash locally
141031-12:28:56,232 workflow DEBUG:
     Setting node inputs
141031-12:28:56,232 workflow DEBUG:
     []
141031-12:28:56,232 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_03/_0x9390918749996889da804dc434301116.json
141031-12:28:56,232 workflow DEBUG:
     Hash exists False
141031-12:28:56,232 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,234 workflow INFO:
     Executing: get_dir_26 ID: 11
141031-12:28:56,234 workflow DEBUG:
     checking hash locally
141031-12:28:56,234 workflow DEBUG:
     Setting node inputs
141031-12:28:56,235 workflow DEBUG:
     []
141031-12:28:56,235 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_26/_0x2f71d52cf29262057aef370b140bf35e.json
141031-12:28:56,235 workflow DEBUG:
     Hash exists False
141031-12:28:56,235 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,237 workflow INFO:
     Executing: get_dir_04 ID: 12
141031-12:28:56,237 workflow DEBUG:
     checking hash locally
141031-12:28:56,237 workflow DEBUG:
     Setting node inputs
141031-12:28:56,237 workflow DEBUG:
     []
141031-12:28:56,237 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_04/_0xcf624b7607ae1bc36213df7f45db4935.json
141031-12:28:56,238 workflow DEBUG:
     Hash exists False
141031-12:28:56,238 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,240 workflow INFO:
     Executing: get_dir_07 ID: 13
141031-12:28:56,240 workflow DEBUG:
     checking hash locally
141031-12:28:56,240 workflow DEBUG:
     Setting node inputs
141031-12:28:56,241 workflow DEBUG:
     []
141031-12:28:56,241 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_07/_0x4e5766a643f0b00234e3ec8139745a11.json
141031-12:28:56,241 workflow DEBUG:
     Hash exists False
141031-12:28:56,241 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,243 workflow INFO:
     Executing: get_dir_24 ID: 14
141031-12:28:56,243 workflow DEBUG:
     checking hash locally
141031-12:28:56,243 workflow DEBUG:
     Setting node inputs
141031-12:28:56,243 workflow DEBUG:
     []
141031-12:28:56,243 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_24/_0x2b2c6bce83bd775188257fbe7d7f016f.json
141031-12:28:56,243 workflow DEBUG:
     Hash exists False
141031-12:28:56,244 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,245 workflow INFO:
     Executing: get_dir_27 ID: 15
141031-12:28:56,245 workflow DEBUG:
     checking hash locally
141031-12:28:56,245 workflow DEBUG:
     Setting node inputs
141031-12:28:56,246 workflow DEBUG:
     []
141031-12:28:56,246 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_27/_0x2d680fa1c9c1a7989080ba9f75b466c1.json
141031-12:28:56,246 workflow DEBUG:
     Hash exists False
141031-12:28:56,246 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,249 workflow INFO:
     Executing: get_dir_28 ID: 16
141031-12:28:56,250 workflow DEBUG:
     checking hash locally
141031-12:28:56,250 workflow DEBUG:
     Setting node inputs
141031-12:28:56,250 workflow DEBUG:
     []
141031-12:28:56,251 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_28/_0x2aaa3f1d82cc0f420e9d2335049b30dd.json
141031-12:28:56,251 workflow DEBUG:
     Hash exists False
141031-12:28:56,251 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,253 workflow INFO:
     Executing: get_dir_02 ID: 17
141031-12:28:56,253 workflow DEBUG:
     checking hash locally
141031-12:28:56,254 workflow DEBUG:
     Setting node inputs
141031-12:28:56,254 workflow DEBUG:
     []
141031-12:28:56,254 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_02/_0x179da965f1e16823a6585b46038df6d5.json
141031-12:28:56,254 workflow DEBUG:
     Hash exists False
141031-12:28:56,255 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,256 workflow INFO:
     Executing: get_dir_22 ID: 18
141031-12:28:56,256 workflow DEBUG:
     checking hash locally
141031-12:28:56,256 workflow DEBUG:
     Setting node inputs
141031-12:28:56,257 workflow DEBUG:
     []
141031-12:28:56,257 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_22/_0x6ae1bb084b43886cde3a14344ad069a0.json
141031-12:28:56,257 workflow DEBUG:
     Hash exists False
141031-12:28:56,257 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,259 workflow INFO:
     Executing: get_dir_01 ID: 19
141031-12:28:56,259 workflow DEBUG:
     checking hash locally
141031-12:28:56,259 workflow DEBUG:
     Setting node inputs
141031-12:28:56,260 workflow DEBUG:
     []
141031-12:28:56,260 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_01/_0xc7ed834bafd6b794ef17cf79b74b2471.json
141031-12:28:56,260 workflow DEBUG:
     Hash exists False
141031-12:28:56,260 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,262 workflow INFO:
     Executing: get_dir_08 ID: 20
141031-12:28:56,262 workflow DEBUG:
     checking hash locally
141031-12:28:56,262 workflow DEBUG:
     Setting node inputs
141031-12:28:56,262 workflow DEBUG:
     []
141031-12:28:56,263 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_08/_0x0c7baf2fc4524ae1247179679e02f2ed.json
141031-12:28:56,263 workflow DEBUG:
     Hash exists False
141031-12:28:56,263 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,265 workflow INFO:
     Executing: get_dir_29 ID: 21
141031-12:28:56,265 workflow DEBUG:
     checking hash locally
141031-12:28:56,265 workflow DEBUG:
     Setting node inputs
141031-12:28:56,265 workflow DEBUG:
     []
141031-12:28:56,265 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_29/_0x0ea7243ae59f1280dd870abc8f89260a.json
141031-12:28:56,266 workflow DEBUG:
     Hash exists False
141031-12:28:56,266 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,267 workflow INFO:
     Executing: get_dir_19 ID: 22
141031-12:28:56,267 workflow DEBUG:
     checking hash locally
141031-12:28:56,267 workflow DEBUG:
     Setting node inputs
141031-12:28:56,268 workflow DEBUG:
     []
141031-12:28:56,268 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_19/_0x3e8d8d16fa66136ce1dd4ec1ea100317.json
141031-12:28:56,268 workflow DEBUG:
     Hash exists False
141031-12:28:56,268 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,270 workflow INFO:
     Executing: get_dir_10 ID: 23
141031-12:28:56,270 workflow DEBUG:
     checking hash locally
141031-12:28:56,270 workflow DEBUG:
     Setting node inputs
141031-12:28:56,270 workflow DEBUG:
     []
141031-12:28:56,271 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_10/_0xf535e3b4e27930cbd4ed09abe566c1df.json
141031-12:28:56,271 workflow DEBUG:
     Hash exists False
141031-12:28:56,271 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,272 workflow INFO:
     Executing: get_dir_12 ID: 24
141031-12:28:56,273 workflow DEBUG:
     checking hash locally
141031-12:28:56,273 workflow DEBUG:
     Setting node inputs
141031-12:28:56,273 workflow DEBUG:
     []
141031-12:28:56,273 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_12/_0x3cbe5858010561a8031e0dfe397628e0.json
141031-12:28:56,273 workflow DEBUG:
     Hash exists False
141031-12:28:56,273 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,275 workflow INFO:
     Executing: get_dir_17 ID: 25
141031-12:28:56,275 workflow DEBUG:
     checking hash locally
141031-12:28:56,275 workflow DEBUG:
     Setting node inputs
141031-12:28:56,276 workflow DEBUG:
     []
141031-12:28:56,276 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_17/_0x5e3d0ac43ff736a574178b6958de2e2c.json
141031-12:28:56,276 workflow DEBUG:
     Hash exists False
141031-12:28:56,276 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,277 workflow INFO:
     Executing: get_dir_14 ID: 26
141031-12:28:56,278 workflow DEBUG:
     checking hash locally
141031-12:28:56,278 workflow DEBUG:
     Setting node inputs
141031-12:28:56,278 workflow DEBUG:
     []
141031-12:28:56,278 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_14/_0x4e6959507658ed796309aa564e9deb5b.json
141031-12:28:56,278 workflow DEBUG:
     Hash exists False
141031-12:28:56,279 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,280 workflow INFO:
     Executing: get_dir_18 ID: 27
141031-12:28:56,280 workflow DEBUG:
     checking hash locally
141031-12:28:56,281 workflow DEBUG:
     Setting node inputs
141031-12:28:56,281 workflow DEBUG:
     []
141031-12:28:56,281 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_18/_0xaa1fb62fa012e1991b7d455bf55f5989.json
141031-12:28:56,281 workflow DEBUG:
     Hash exists False
141031-12:28:56,281 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,283 workflow INFO:
     Executing: get_dir_16 ID: 28
141031-12:28:56,283 workflow DEBUG:
     checking hash locally
141031-12:28:56,283 workflow DEBUG:
     Setting node inputs
141031-12:28:56,284 workflow DEBUG:
     []
141031-12:28:56,284 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_16/_0xd14612b2cd5ed014379b8bfc81f0c1dd.json
141031-12:28:56,284 workflow DEBUG:
     Hash exists False
141031-12:28:56,284 workflow DEBUG:
     Finished checking hash True
141031-12:28:56,285 workflow INFO:
     Executing: get_dir_20 ID: 29
141031-12:28:56,286 workflow DEBUG:
     checking hash locally
141031-12:28:56,286 workflow DEBUG:
     Setting node inputs
141031-12:28:56,286 workflow DEBUG:
     []
141031-12:28:56,286 workflow DEBUG:
     /tmp/deweybe/test/test_ipy/get_dir_20/_0x10adcd933f19eb98f4338f74ea482353.json
141031-12:28:56,286 workflow DEBUG:
     Hash exists False
141031-12:28:56,287 workflow DEBUG:
     Finished checking hash True
141031-12:28:58,317 workflow INFO:
     [Job finished] jobname: get_dir_25 jobid: 0
141031-12:28:58,320 workflow DEBUG:
     Clearing id: 1
141031-12:28:58,322 workflow INFO:
     [Job finished] jobname: get_dir_00 jobid: 1
141031-12:28:58,324 workflow DEBUG:
     Clearing id: 2
141031-12:28:58,326 workflow INFO:
     [Job finished] jobname: get_dir_05 jobid: 2
141031-12:28:58,327 workflow DEBUG:
     Clearing id: 3
141031-12:28:58,329 workflow INFO:
     [Job finished] jobname: get_dir_06 jobid: 3
141031-12:28:58,331 workflow DEBUG:
     Clearing id: 4
141031-12:28:58,333 workflow INFO:
     [Job finished] jobname: get_dir_23 jobid: 4
141031-12:28:58,334 workflow DEBUG:
     Clearing id: 5
141031-12:28:58,336 workflow INFO:
     [Job finished] jobname: get_dir_15 jobid: 5
141031-12:28:58,338 workflow DEBUG:
     Clearing id: 6
141031-12:28:58,340 workflow INFO:
     [Job finished] jobname: get_dir_21 jobid: 6
141031-12:28:58,341 workflow DEBUG:
     Clearing id: 7
141031-12:28:58,343 workflow INFO:
     [Job finished] jobname: get_dir_09 jobid: 7
141031-12:28:58,345 workflow DEBUG:
     Clearing id: 8
141031-12:28:58,347 workflow INFO:
     [Job finished] jobname: get_dir_11 jobid: 8
141031-12:28:58,348 workflow DEBUG:
     Clearing id: 9
141031-12:28:58,350 workflow INFO:
     [Job finished] jobname: get_dir_13 jobid: 9
141031-12:28:58,352 workflow DEBUG:
     Clearing id: 10
141031-12:28:58,354 workflow INFO:
     [Job finished] jobname: get_dir_03 jobid: 10
141031-12:28:58,355 workflow DEBUG:
     Clearing id: 11
141031-12:28:58,357 workflow INFO:
     [Job finished] jobname: get_dir_26 jobid: 11
141031-12:28:58,359 workflow DEBUG:
     Clearing id: 12
141031-12:28:58,361 workflow INFO:
     [Job finished] jobname: get_dir_04 jobid: 12
141031-12:28:58,362 workflow DEBUG:
     Clearing id: 13
141031-12:28:58,364 workflow INFO:
     [Job finished] jobname: get_dir_07 jobid: 13
141031-12:28:58,366 workflow DEBUG:
     Clearing id: 14
141031-12:28:58,367 workflow INFO:
     [Job finished] jobname: get_dir_24 jobid: 14
141031-12:28:58,369 workflow DEBUG:
     Clearing id: 15
141031-12:28:58,371 workflow INFO:
     [Job finished] jobname: get_dir_27 jobid: 15
141031-12:28:58,372 workflow DEBUG:
     Clearing id: 16
141031-12:28:58,374 workflow INFO:
     [Job finished] jobname: get_dir_28 jobid: 16
141031-12:28:58,376 workflow DEBUG:
     Clearing id: 17
141031-12:28:58,378 workflow INFO:
     [Job finished] jobname: get_dir_02 jobid: 17
141031-12:28:58,380 workflow DEBUG:
     Clearing id: 18
141031-12:28:58,382 workflow INFO:
     [Job finished] jobname: get_dir_22 jobid: 18
141031-12:28:58,383 workflow DEBUG:
     Clearing id: 19
141031-12:28:58,385 workflow INFO:
     [Job finished] jobname: get_dir_01 jobid: 19
141031-12:28:58,386 workflow DEBUG:
     Clearing id: 20
141031-12:28:58,388 workflow INFO:
     [Job finished] jobname: get_dir_08 jobid: 20
141031-12:28:58,390 workflow DEBUG:
     Clearing id: 21
141031-12:28:58,392 workflow INFO:
     [Job finished] jobname: get_dir_29 jobid: 21
141031-12:28:58,393 workflow DEBUG:
     Clearing id: 22
141031-12:28:58,395 workflow INFO:
     [Job finished] jobname: get_dir_19 jobid: 22
141031-12:28:58,397 workflow DEBUG:
     Clearing id: 23
141031-12:28:58,398 workflow INFO:
     [Job finished] jobname: get_dir_10 jobid: 23
141031-12:28:58,400 workflow DEBUG:
     Clearing id: 24
141031-12:28:58,402 workflow INFO:
     [Job finished] jobname: get_dir_12 jobid: 24
141031-12:28:58,403 workflow DEBUG:
     Clearing id: 25
141031-12:28:58,405 workflow INFO:
     [Job finished] jobname: get_dir_17 jobid: 25
141031-12:28:58,407 workflow DEBUG:
     Clearing id: 26
141031-12:28:58,409 workflow INFO:
     [Job finished] jobname: get_dir_14 jobid: 26
141031-12:28:58,410 workflow DEBUG:
     Clearing id: 27
141031-12:28:58,412 workflow INFO:
     [Job finished] jobname: get_dir_18 jobid: 27
141031-12:28:58,414 workflow DEBUG:
     Clearing id: 28
141031-12:28:58,416 workflow INFO:
     [Job finished] jobname: get_dir_16 jobid: 28
141031-12:28:58,417 workflow DEBUG:
     Clearing id: 29
141031-12:28:58,419 workflow INFO:
     [Job finished] jobname: get_dir_20 jobid: 29
141031-12:28:58,420 workflow DEBUG:
     Clearing id: 30
141031-12:28:58,422 workflow DEBUG:
     Number of pending tasks: 0
141031-12:28:58,422 workflow DEBUG:
     Slots available: None
141031-12:29:00,428 workflow DEBUG:
     Aggregate: False
141031-12:29:00,431 workflow DEBUG:
     Aggregate: False
141031-12:29:00,434 workflow DEBUG:
     Aggregate: False
141031-12:29:00,437 workflow DEBUG:
     Aggregate: False
141031-12:29:00,440 workflow DEBUG:
     Aggregate: False
141031-12:29:00,443 workflow DEBUG:
     Aggregate: False
141031-12:29:00,446 workflow DEBUG:
     Aggregate: False
141031-12:29:00,448 workflow DEBUG:
     Aggregate: False
141031-12:29:00,451 workflow DEBUG:
     Aggregate: False
141031-12:29:00,454 workflow DEBUG:
     Aggregate: False
141031-12:29:00,457 workflow DEBUG:
     Aggregate: False
141031-12:29:00,460 workflow DEBUG:
     Aggregate: False
141031-12:29:00,463 workflow DEBUG:
     Aggregate: False
141031-12:29:00,466 workflow DEBUG:
     Aggregate: False
141031-12:29:00,468 workflow DEBUG:
     Aggregate: False
141031-12:29:00,471 workflow DEBUG:
     Aggregate: False
141031-12:29:00,474 workflow DEBUG:
     Aggregate: False
141031-12:29:00,477 workflow DEBUG:
     Aggregate: False
141031-12:29:00,480 workflow DEBUG:
     Aggregate: False
141031-12:29:00,483 workflow DEBUG:
     Aggregate: False
141031-12:29:00,486 workflow DEBUG:
     Aggregate: False
141031-12:29:00,488 workflow DEBUG:
     Aggregate: False
141031-12:29:00,491 workflow DEBUG:
     Aggregate: False
141031-12:29:00,494 workflow DEBUG:
     Aggregate: False
141031-12:29:00,497 workflow DEBUG:
     Aggregate: False
141031-12:29:00,500 workflow DEBUG:
     Aggregate: False
141031-12:29:00,503 workflow DEBUG:
     Aggregate: False
141031-12:29:00,506 workflow DEBUG:
     Aggregate: False
141031-12:29:00,509 workflow DEBUG:
     Aggregate: False
141031-12:29:00,512 workflow DEBUG:
     Aggregate: False
/tmp/deweybe/test/test_ipy/get_dir_20
/tmp/deweybe/test/test_ipy/get_dir_16
/tmp/deweybe/test/test_ipy/get_dir_18
/tmp/deweybe/test/test_ipy/get_dir_14
/tmp/deweybe/test/test_ipy/get_dir_17
/tmp/deweybe/test/test_ipy/get_dir_12
/tmp/deweybe/test/test_ipy/get_dir_10
/tmp/deweybe/test/test_ipy/get_dir_19
/tmp/deweybe/test/test_ipy/get_dir_29
/tmp/deweybe/test/test_ipy/get_dir_08
/tmp/deweybe/test/test_ipy/get_dir_01
/tmp/deweybe/test/test_ipy/get_dir_22
/tmp/deweybe/test/test_ipy/get_dir_02
/tmp/deweybe/test/test_ipy/get_dir_28
/tmp/deweybe/test/test_ipy/get_dir_27
/tmp/deweybe/test/test_ipy/get_dir_24
/tmp/deweybe/test/test_ipy/get_dir_07
/tmp/deweybe/test/test_ipy/get_dir_04
/tmp/deweybe/test/test_ipy/get_dir_26
/tmp/deweybe/test/test_ipy/get_dir_03
/tmp/deweybe/test/test_ipy/get_dir_13
/tmp/deweybe/test/test_ipy/get_dir_11
/tmp/deweybe/test/test_ipy/get_dir_09
/tmp/deweybe/test/test_ipy/get_dir_21
/tmp/deweybe/test/test_ipy/get_dir_15
/tmp/deweybe/test/test_ipy/get_dir_23
/tmp/deweybe/test/test_ipy/get_dir_06
/tmp/deweybe/test/test_ipy/get_dir_05
/tmp/deweybe/test/test_ipy/get_dir_00
/tmp/deweybe/test/test_ipy/get_dir_25

[miykael]

@satra - I've run the code with 1000 nodes, but nothing happened. But right now my other code which sometimes causes the error is also doing fine. But I'm eager to test your snippet again as soon as my system causing trouble again.