🔧 slurm batch system improvements

[juanesarango]

Improvements:

• Use bash instead of sh
• SLURM doesn't inherit the ENV variables by default. For most jobs we need to pass along the conventional $TMP, $TMPDIR and/or $TMP_DIR vars.
• LSF had a nice feature of printing the job metrics at the end of each job. Slurm doesn't. But it has an utility seff <jobid> to do so. Here I add an extra job that is run after the main job finishes and stores the metrics info in a new head_job.slurm file.

Type of PR:

• 🐛   Bug fix
• 🚀   New feature
• 🔧   Code refactor
• ⚠️   Breaking change that would cause existing functionality to change
• 📘   I have updated the documentation accordingly
• ✅   I have added tests to cover my changes

[ddomenico]

Big fan of storing the metrics like this 👍 do you have an example of a job run with this?

[ddomenico]

Is the second bash here redundant?

[juanesarango]

One is for the shebang. The other one is for execute the file.

#!/bin/bash

bash <command>

I guess it could work without it. as long as the file {root}/in.$SLURM_ARRAY_TASK_ID has the proper shebang too.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.13%. Comparing base (e030016) to head (f743e01).
Report is 4 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master      #62      +/-   ##
==========================================
+ Coverage   95.12%   95.13%   +0.01%     
==========================================
  Files          19       19              
  Lines        2359     2365       +6     
==========================================
+ Hits         2244     2250       +6     
  Misses        115      115              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[juanesarango]

Big fan of storing the metrics like this 👍 do you have an example of a job run with this?

Yeah, check this one:

$ cat /data1/papaemme/isabl/data/analyses/22/26/482226/head_job.slurm
Job ID: 1094711
Array Job ID: 1094711_1
Cluster: iris
User/Group: svc_papaemme_bot/svc_papaemme_bot
State: COMPLETED (exit code 0)
Cores: 1
CPU Utilized: 00:00:53
CPU Efficiency: 65.43% of 00:01:21 core-walltime
Job Wall-clock time: 00:01:21
Memory Utilized: 1.59 GB
Memory Efficiency: 1.59% of 100.00 GB