FIX: SLURM plugin polling

[mgxd]

Summary

Related to #1853 .

List of changes proposed in this PR (pull-request)

• Catch slurm squeue error when jobid has completed/terminated

Acknowledgment

• (Mandatory) I acknowledge that this contribution will be available under the Apache 2 license.

[mgxd]

@agt24 @atsuch

could you test your workflow(s) with this branch and let us know if things are running smoothly?

[effigies]

Based on #1853 (comment), I think this should specifically be RuntimeError?

Otherwise, this looks reasonable.

[effigies]

@agt24 @atsuch Have you had a chance to test out this branch?

[effigies]

This looks reasonable. If we don't hear back by the end of the week, I'm okay merging, and we can see if upgrading to 1.1.3 fixes people's SLURM problems.

[agt24]

@dalejn tried to run this on Friday, but it crashed. Here's the error:
`
Traceback (most recent call last):

File "/data/zhoud4/python/envs/NiPypeUpdate/lib/python3.5/site-packages/nipype/pipeline/plugins/base.py", line 154, in run
result = self._get_result(taskid)

File "/data/zhoud4/python/envs/NiPypeUpdate/lib/python3.5/site-packages/nipype/pipeline/plugins/base.py", line 462, in _get_result
if self._is_pending(taskid):

File "/data/zhoud4/python/envs/NiPypeUpdate/lib/python3.5/site-packages/nipype/pipeline/plugins/slurm.py", line 74, in _is_pending
raise(e)

File "/data/zhoud4/python/envs/NiPypeUpdate/lib/python3.5/site-packages/nipype/pipeline/plugins/slurm.py", line 70, in _is_pending
terminal_output='allatonce').run()

File "/data/zhoud4/python/envs/NiPypeUpdate/lib/python3.5/site-packages/nipype/interfaces/base/core.py", line 521, in run
runtime = self._run_interface(runtime)

File "/data/zhoud4/python/envs/NiPypeUpdate/lib/python3.5/site-packages/nipype/interfaces/base/core.py", line 1033, in _run_interface
self.raise_exception(runtime)

File "/data/zhoud4/python/envs/NiPypeUpdate/lib/python3.5/site-packages/nipype/interfaces/base/core.py", line 970, in raise_exception
).format(**runtime.dictcopy()))

RuntimeError: Command:
squeue -j 9448406
Standard output:

Standard error:
slurm_load_jobs error: Socket timed out on send/recv operation
Return code: 1
`

[mgxd]

@agt24 that error looks like your SLURM master is getting overloaded with requests, are there a surplus of jobs currently queued or running through SLURM?

We should still merge this patch in for 1.1.3

[agt24]

Yes, there are limits on how many jobs a single user can have in the queue. This is likely the cause.

@dalejn will try again with max_jobs set to something manageable.

Agree that the patch can be merged.

[effigies]

Sounds good. Merging. If there's something else sensible to be done on overload (Better error message? Wait and retry?), please open a new issue.