Slurm jobs submissions

[azzaea]

Hello,

I would like to report a problem when submitting jobs to a slurm-scheduled system. On my attempts working with the most recent V1.3 Swift/T, I always get the sbatch failed! message whenever I provoke a hypothetical tst.swift program (or its compiled version).
Here is an example to demonstrate:

$ cat settings.sh 
export QUEUE=normal 
export PPN=4
###
 $ swift-t -m slurm -l -n 3 -s settings.sh tst.swift
TURBINE-SLURM SCRIPT
NODES=3
PROCS=3
PPN=1
sourcing: settings.sh
done: settings.sh
TURBINE_OUTPUT=/home1/04525/tg838247/turbine-output/2017/08/05/19/42/16
TURBINE_HOME=/home1/04525/tg838247/software/swift-t/swift-t-1.3_no_mpe/turbine
wrote: /home1/04525/tg838247/turbine-output/2017/08/05/19/42/16/turbine-slurm.sh
sbatch failed!

Now, the code stored in ${TURBINE_OUTPUT}/submit.txt does actually work on its own (i.e. when it is manually run, it does the right thing: reserving the right memory, time, ..etc and also implements the right swift/t code).
Looking somewhat closely at the code in scripts/submit/slurm/turbine-slurm-run.zsh, this meant jobs were not getting IDs, because they were never submitted to the scheduler to begin with, which can be seen in line 75. Here is a quick attempt to rectify:

JOB_ID=$( ${SUBMIT_COMMAND} | grep "[1-9][0-9]*$" )

This solution is half working: it does submit the swift/t job, and reserves the right resources, but doesn't properly extract the JOB_ID. It is kind of tolerable however, as I can squeue anyway.

I know there must be a neater way to do this, so I'm eager to know it, but I hope this is nonetheless helpful.

Thank you,
Azza

[azzaea]

It seems I was closer than I thought!
The -o option is what was missing:

JOB_ID=$( echo $( $SUBMIT_COMMAND )| grep -o "[1-9][0-9]*$" )

[j-woz]

Super, I will put this in. Out of curiosity can you post the output of $SUBMIT_COMMAND on your system?

[j-woz]

Is the echo necessary or can we do:
JOB_ID=$( $SUBMIT_COMMAND | grep -o "[1-9][0-9]*$" )

[azzaea]

Great timing! I was about to ask about something relevant.

First, here is the output from the $SUBMIT_COMMAND (also stored in the submit.txt file on TURBINE_OUTPUT directory).

sbatch --output=/home1/04525/tg838247/turbine-output/2017/08/07/04/12/23/output.txt --error=/home1/04525/tg838247/turbine-output/2017/08/07/04/12/23/output.txt --partition=normal --job-name=SWIFT /home1/04525/tg838247/turbine-output/2017/08/07/04/12/23/turbine-slurm.sh

Good point about the echo command. Submission and checking happens smoothly without it, so no need.

Now, my question. I'm not sure how the resources are allocated for different invocations of my program.
In particular, I'm referring to the output.txt (which should be provenance of the run) and turbine.log (which should give the allocated resources) files that are generated after a successful job submission. My understanding is that PROCS specifies the number of processors (i.e. mpi ranks) for the job, PPN the number of processes per node, and NODES the number of physical nodes or machines to be reserved for the job; and that the math is basically: NODES=PROCS/PPN

Below are some examples where these number are different between these 2 files with different settings:

• I only specify export PROCS=12 (so, I'm expecting 12 mpi ranks), yet I get only 2 ranks on 2 nodes:

 $ cat /home1/04525/tg838247/turbine-output/2017/08/07/08/18/10/output.txt 
TURBINE-SLURM.SH
ADLB_DEBUG_RANKS: rank: 0 nodename: c402-133.stampede2.tacc.utexas.edu
ADLB_DEBUG_RANKS: rank: 1 nodename: c403-073.stampede2.tacc.utexas.edu
   0.000 WORK TYPES: WORK
   0.001 WORKERS: 1 RANKS: 0 - 0
   0.001 SERVERS: 1 RANKS: 1 - 1
   0.001 WORK WORKERS: 1 RANKS: 0 - 0

$ cat /home1/04525/tg838247/turbine-output/2017/08/07/08/18/10/turbine.log 
SCRIPT:            ./swift-t-tst.eCK.tic
JOB:               169072
COMMAND:           //bin/tclsh8.5 /home1/04525/tg838247/turbine-output/2017/08/07/08/18/10/swift-t-tst.eCK.tic 
TURBINE_OUTPUT:    /home1/04525/tg838247/turbine-output/2017/08/07/08/18/10
HOSTNAME:          stampede2.tacc.utexas.edu
SUBMITTED:         08/07/2017 08:18AM
PROCS:             12
NODES:             2
PPN:               1
TURBINE_WORKERS:   1
ADLB_SERVERS:      1
WALLTIME:          01:00:00
ADLB_EXHAUST_TIME: 1

Now, when I also set export PPN=3 (so expecting 12 ranks, 3 ranks per node ==> 4 nodes), the allocation is 6 ranks on 2 nodes

 $ cat /home1/04525/tg838247/turbine-output/2017/08/07/08/20/01/output.txt 
TURBINE-SLURM.SH
ADLB_DEBUG_RANKS: rank: 0 nodename: c402-133.stampede2.tacc.utexas.edu
ADLB_DEBUG_RANKS: rank: 5 nodename: c403-073.stampede2.tacc.utexas.edu
ADLB_DEBUG_RANKS: rank: 2 nodename: c402-133.stampede2.tacc.utexas.edu
ADLB_DEBUG_RANKS: rank: 1 nodename: c402-133.stampede2.tacc.utexas.edu
ADLB_DEBUG_RANKS: rank: 3 nodename: c403-073.stampede2.tacc.utexas.edu
ADLB_DEBUG_RANKS: rank: 4 nodename: c403-073.stampede2.tacc.utexas.edu
   0.001 WORK TYPES: WORK
   0.001 WORKERS: 5 RANKS: 0 - 4
   0.001 SERVERS: 1 RANKS: 5 - 5
   0.001 WORK WORKERS: 5 RANKS: 0 - 4

$ cat /home1/04525/tg838247/turbine-output/2017/08/07/08/20/01/turbine.log 
SCRIPT:            ./swift-t-tst.Gj5.tic
JOB:               169074
COMMAND:           //bin/tclsh8.5 /home1/04525/tg838247/turbine-output/2017/08/07/08/20/01/swift-t-tst.Gj5.tic 
TURBINE_OUTPUT:    /home1/04525/tg838247/turbine-output/2017/08/07/08/20/01
HOSTNAME:          stampede2.tacc.utexas.edu
SUBMITTED:         08/07/2017 08:20AM
PROCS:             12
NODES:             2
PPN:               3
TURBINE_WORKERS:   1
ADLB_SERVERS:      1
WALLTIME:          01:00:00
ADLB_EXHAUST_TIME: 1

And finally, here is another test, this time adding to the setting export NODES=4 (so expecting 12 mpi ranks, 4 nodes, 3 ranks each); yet getting 12 ranks on 4 nodes

$ cat /home1/04525/tg838247/turbine-output/2017/08/07/09/25/36/output.txt 
TURBINE-SLURM.SH
ADLB_DEBUG_RANKS: rank: 6 nodename: c427-074.stampede2.tacc.utexas.edu
ADLB_DEBUG_RANKS: rank: 3 nodename: c426-072.stampede2.tacc.utexas.edu
ADLB_DEBUG_RANKS: rank: 0 nodename: c413-021.stampede2.tacc.utexas.edu
ADLB_DEBUG_RANKS: rank: 7 nodename: c427-074.stampede2.tacc.utexas.edu
ADLB_DEBUG_RANKS: rank: 4 nodename: c426-072.stampede2.tacc.utexas.edu
ADLB_DEBUG_RANKS: rank: 9 nodename: c427-081.stampede2.tacc.utexas.edu
ADLB_DEBUG_RANKS: rank: 2 nodename: c413-021.stampede2.tacc.utexas.edu
ADLB_DEBUG_RANKS: rank: 8 nodename: c427-074.stampede2.tacc.utexas.edu
ADLB_DEBUG_RANKS: rank: 5 nodename: c426-072.stampede2.tacc.utexas.edu
ADLB_DEBUG_RANKS: rank: 10 nodename: c427-081.stampede2.tacc.utexas.edu
ADLB_DEBUG_RANKS: rank: 1 nodename: c413-021.stampede2.tacc.utexas.edu
ADLB_DEBUG_RANKS: rank: 11 nodename: c427-081.stampede2.tacc.utexas.edu
   0.001 WORK TYPES: WORK
   0.001 WORKERS: 11 RANKS: 0 - 10
   0.001 SERVERS: 1 RANKS: 11 - 11
   0.001 WORK WORKERS: 11 RANKS: 0 - 10

[azzaea]

Hello again,

As a follow up, I'm wondering if there is a way to map directly between Swift-t options, and standard slurm options. I'm trying to run a pipeline with the settings below:

export PROCS=10 #total number of MPI processes
export PPN=2 #Number of processes per node 
export NODES=3

Yet, the scheduler records: You are requesting 408 Physical cores on 3 nodes. What is it I'm missing here?

Thank you,
Azza

[j-woz]

Moving this to a new issue #138 .