CRAWLER2

                                      Adiv Paradise
                                   University of Toronto


======================================README=====================================

Prerequisites: Python, NumPy.

This program is a "shadow scheduler" for Sunnyvale (though it can easily be 
adapted for other systems). The intent is to manage your job submissions in
an automated manner, thereby increasing the number of jobs you run and reducing
the amount of work you have to do, without gumming up the queue for several months. 
Once you hit a limit you define, new jobs wait until old jobs finish before being
*submitted*, so other people can get in line too. This is 'ethical' job 
scheduling. Crawler2 does not live in virtual memory like most programs; instead it 
resides in disk memory like any normal file. This is how it is able to circumvent 
walltime limits--from the cluster's perspective, this program is almost never 
running! We accomplish this by making sure that everything the program needs to 
pick up where it left off is written to file in a form and location that is 
accessible to the program. When the program is executed, it learns where it 
should leave off from those files, and then picks up as if it had been running 
the whole time.

Main components:

crawler2.py
---------------------
This is the main program, and what you should execute to get the program going.
It reads the file tasks.crwl, which contains recipes for the various jobs you want
to run. It checks to see how many jobs are currently running according to the
various running_*.crwl files, and checks to see how many nodes you want to be
using according to nnodes.crwl. It will then prepare and submit as many jobs as 
necessary to get you up to that limit. If there are no more jobs to submit, or 
you are already at your limit, it will peacefully exit.


crawlset.py
---------------------
This is simply one side of the interface between crawler2 and your models. You
should pretty much never have to concern yourself with this file.


identity.py (IMPORTANT)
---------------------
This holds on to just a few variables, which the program will use to tell the
scheduler who you are. This is kept in a separate file for module inheritance
purposes. You should modify this file to have the appropriate username. For 
PBS Torque, only the username matters, but for Slurm you'll need to change the
email address and account as well.


batch_system.py (IMPORTANT)
--------------------
Also an extremely simple file: this just importants the correct scheduler module.
Several other modules (including the program hooks) import this, which is why it
exists--so that you only need to change the importation line in one place.


crawldefs.py
---------------------
This file is integral: it defines the Job object, which stores information about
a job in way that's convenient for the program to use, and also defines the
global variables MODELS and USER. You should change these to reflect your 
Sunnyvale username and the models you want to use. For each model, you'll need
to specify how many jobs of that kind can run simultaneously on an 8-cpu node.
For example, if you have a code that is parallelized to run on 8 cores, then
its value in this dictionary is 1. You can only run one instance at a time per node.
But if you have a code that runs in serial, with no parallelism, then its value
is 8, since you can have 8 going at once on a single node.


torque.py/slurm.py/etc (Will probably have to be modified)
----------------------------------------------------------
These modules tell crawler how to manage jobs on your cluster. I've included
modules for the PBS TORQUE system in place on the CITA Sunnyvale cluster, as 
well as one for the Slurm system in place on the University of Chicago's 
Midway2 cluster. However, implementations may vary, so you should take a 
careful look and make sure that job items will be read correctly. In particular,
how one queries job info (qstat -f <ID>) may vary, including the format <ID>
needs to be in. This is likely to be the component that gives you the most 
headaches. If you use any of my example set<model>.py files, you will 
almost certainly need to change which modules get loaded on the cluster.

Once you modify or write your own <scheduler.py> file, have batch_system.py
import everything from it ("from <scheduler> import *").


tasks.crwl (IMPORTANT)
----------------------------
This is the hippocampus of the program. It is the list of all tasks for the
program to queue. Each line is either a header or a job. The first five items
on a given line have a set format that you cannot change, but after that it is
entirely up to you--the header fields and arguments in the job line will just
be passed to the model interface you write. 

                    tasks.crwl header format:
# PID MODEL JOBNAME STATUS NCORES QUEUE arg1 arg2 arg3 etc etc
    
    -> # is a marker indicating that this line is a header line.

Below the header line, you can include an arbitrary number of job lines, so
long as they all have the same number of items as the header (sans the # character).
When the program reads a job line, it then backtracks up the list until it 
finds a header line with the right number of arguments. 
                    
                    Creating a job line:
    -> Do not include anything for '#'. The first item should be the PID.
    -> MODEL is the name of the model you want to run for this job
    -> JOBNAME is the name of the job which will appear in the queue
    -> STATUS is the job's completion status. Set it to 0 initially. It will
          change to 1 when the job is submitted, and 2 when the job is done.
    -> PID is the ID number for the job. The first one in the list should be
          1, and then they should increase sequentially from there.
    -> NCORES is the number of CPUs your job will use. The program uses this
          to determine how much of your desired allocation this job will use.
    -> QUEUE: this one will have to be handled by your set<model>.py routine,
          but is intended to identify which submission queue this job will use.

The last line of the file should be blank. 


priority.crwl (IMPORTANT)
----------------------------
This is almost identical to tasks.crwl, with two major differences: first,
PIDs should be negative, starting with -1. Second, the purpose of this file
is to queue up jobs that you want done as soon as possible, perhaps because
you examined output from runs in tasks.crwl and need immediate follow-up. Any 
open jobs in priority.crwl will be queued before any new jobs from tasks.crwl. 
You should *not* use priority.crwl as the main list of jobs to queue, because
that will leave you with no way to inject jobs with a higher priority than
the main queue.


nnodes.crwl (IMPORTANT)
--------------------------
The number of 8-cpu nodes you'd like to use. Note that most nodes actually 
have more than 8 cpus these days.


inuse.crwl
--------------------------
0 or 1, to indicate whether tasks.crwl is currently in use. You should set 
this to 1 when adding jobs to avoid colliding with an exiting job. When jobs
exit, they make use of this file to make sure only one instance of crawler2.py
is running.


waitlist (folder)
--------------------------
When programs exit, they do some cleanup work and then execute the main program.
If the main program is already in use (inuse.crwl is set to 1), they place a 
unique token in this folder, and then quietly exit. If it is not in use, they
will claim ownership (set inuse.crwl to 1), and (in theory) unpack all tokens 
in this folder, extracting the unique job information necessary to identify the
job in tasks.crwl or priority.crwl. They will then set the job statuses to 2
to indicate they have completed, and clear the token from the folder.


release.py
--------------------------
This is a crucial component, but you shouldn't need to modify it. You DO however
need to make sure that at the end of each job, release.py is executed. This 
tells the main program that this job is no longer running, that it has completed,
and then if the coast is clear, executes the main program to make sure another 
job is submitted in its place. If not, it will place a unique token in the waitlist.
This component is why once you run crawler2.py the first time, it will keep itself 
going for as long as there are unfinished jobs in tasks.crwl or priority.crwl.


hopper (folder)
--------------------------
This is a useful general-purpose folder, whose use is not required, but recommended.
You can put useful input or boundary condition files here, and tell your codes to
fetch those files as parameters in tasks.crwl. You can also use this to implement
software pipelines, where one model produces output, which is placed in the hopper,
and which is then used as input to another code (such as a postprocessor).


set<model>.py (CRUCIAL)
------------------------------
This is the most important part of the whole apparatus, and it's one you have to
write. This is the interface to your model. There are two rules: it must have a 
function called 'prep' that takes a Job object as an argument, and it must have 
a 'submit' function that also takes a Job object as an argument. The Job object 
includes the header fields for the job and the arguments for the job, stored in 
a dictionary, job.parameters. You can access each argument (as a string) via 
job.parameters[field], where field is one of the options you want to support in 
the header line. 

The prep function should read all the arguments, and use them to construct the 
job you want to run, as if it was going to be run immediately: copy the 
executable/model into a work directory, change input or namelist files, modify 
boundary conditions, and finally, write the submission script. Remember that at 
the end of the job, you must execute release.py with "python release.py". 

The submit function should 'cd' into the work directory, submit the job using
the submission script, and 'cd' back to the main directory. This function is
separated from the job preparation routine to provide 'dry run' functionality:
you can run the program so that it prepares jobs but doesn't submit them, so you
can check to make sure the jobs have been set up correctly. To use this mode,
run the main program with "python crawler2.py DRYRUN".

Several example interfaces have been provided, which I use for my own codes.
The SBDART interface is fully-operational in its LMDZ and PlaSim modes, and 
the Postprocess interface is also fully-operational (and much more minimal).
The PlaSim interface is the most fully-functional and complex interface yet.

One last thing: create a folder for each model, named <model>. Within that folder, 
you should include a folder with any files you need to run your code, and which your 
set<model>.py routine will access to construct each run. A good practice is to 
call this folder "clean"--that clearly indicates that it contains a clean version
of the code, and allows you to create alternative folders for modified codes. 
setplasim.py is an example of an interface that supports multiple source folders.
The program will create run directories in that folder no matter what to store 
information about current jobs running that model, even if you choose to put 
the work directories elsewhere (as with the Postprocess interface). You should also
create a folder called <model>output, in case your specialized code doesn't create
it automatically. Model outputs will go in <model>/output.