CI: Add retry script for Docker commands

[effigies]

Fixes #2489.

Changes proposed in this pull request

• Add retry_cmd.sh script
• Replace loops in .circleci/config.yml

[codecov-io]

Codecov Report

Merging #2516 into master will increase coverage by 0.04%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2516      +/-   ##
==========================================
+ Coverage   66.84%   66.88%   +0.04%     
==========================================
  Files         327      327              
  Lines       42479    42479              
  Branches     5268     5268              
==========================================
+ Hits        28393    28411      +18     
+ Misses      13381    13366      -15     
+ Partials      705      702       -3

Flag	Coverage Δ
#smoketests	50.83% <ø> (+0.3%)	⬆️
#unittests	64.22% <ø> (-0.01%)	⬇️

Impacted Files	Coverage Δ
nipype/pipeline/engine/nodes.py	83.91% <0%> (+0.33%)	⬆️
nipype/utils/provenance.py	84.71% <0%> (+1.27%)	⬆️
nipype/workflows/fmri/spm/preprocess.py	72.44% <0%> (+12.24%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ab3ea2d...2cd3da3. Read the comment docs.

[effigies]

There are some limits to this script WRT multi-command behavior.

$ tools/retry_cmd.sh -n 2 -s 1 "echo TEST && exit 2"; echo $?  
sh: echo TEST && exit 2: command not found
sh: echo TEST && exit 2: command not found
127
$ tools/retry_cmd.sh -n 2 -s 1 echo TEST "&&" exit 2; echo $? 
TEST && exit 2
0

But for the current purposes this seems fine.

[oesteban]

This is a lot more readable than before. Please check the comment I left above.

[oesteban]

Doesn't this need to set set +o pipefail here to work? After some digging, I think the original problem reported in #2489 is just that the steps are run with set -eo pipefile (circle has started to run it with 2.0). Using both -eo will make the script to exit even with that || true pipe.

[effigies]

No, because those apply to each instance of the shell. It doesn't infect subprocesses.