nested controller groups

[yonicd]

How would I go about mimicking the future::plan objects where I can set

future::plan( list(multisession, future.batchtools::batchtools_sge) )

that would manage calls to sge from child processes locally.

My current use case is that I am on a shared node on a cluster (node1) with multiple users and need to send jobs to another cluster where i will need to ssh into a master node (node2) that has a slurm scheduler and then have the controller manage for me the jobs on the node2 and send the results back to node1

Making things more interesting the jobs I am sending to the slurm queue on node2 are stan models so i would need to be able to manage resources for the jobs to have 4 cores per job.

similar to

sge <- future::tweak(
 future.batchtools::batchtools_sge,
 label = 'test',
 template = batchtools.sge-local.tmpl',
 workers = 3,
 resources = list(slots = 4)
 )
future::plan( list(multisession, sge))

batchtools.sge-local.tmpl

#!/bin/bash
## The name of the job, can be anything, simply used when displaying the
list of running jobs
#$ -N <%= job.name %
## Combining output/error messages into one file
#$ -j y
## Giving the name of the output log file
#$ -o <%= log.file %>
## One needs to tell the queue system to use the current directory as the
working directory
## Or else the script may fail as it will execute in your top level home
directory /home/username
#$ -cwd
## Use environment variables
#$ -V
## Use correct queue
#$ -q all.q
<%= if(!is.null(resources[["slots"]])){
 sprintf('#$ -pe smp %s',resources[["slots"]])
}else{
''
}%>
## Export value of DEBUGME environemnt var to slave
export DEBUGME=<%= Sys.getenv("DEBUGME") %>
<%= sprintf("export OMP_NUM_THREADS=%i", resources$omp.threads) -%>
<%= sprintf("export OPENBLAS_NUM_THREADS=%i", resources$blas.threads) -%>
<%= sprintf("export MKL_NUM_THREADS=%i", resources$blas.threads) -%>
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
exit 0

[wlandau]

How would I go about mimicking the future::plan objects where I can set future::plan(list(multisession, future.batchtools::batchtools_sge))

crew has no direct support for nested parallelism like future does. In the more typical case, I recommend that the task submitted to controller$push() manually set up whatever inner parallelism it needs, which could be in the form of another controller.

My current use case is that I am on a shared node on a cluster (node1) with multiple users and need to send jobs to another cluster where i will need to ssh into a master node (node2) that has a slurm scheduler and then have the controller manage for me the jobs on the node2 and send the results back to node1

This sounds most comparable to the SSH connector in clustermq. Security folks in our industry are increasingly cracking down on local-machine-to-login-node SSH tunneling, which is the most typical use for this, so I haven't implemented it in crew either. For your case, you might be able to hack a workaround in crew_controller_slurm() on node1: manually set the port, set up port forwarding from node1 to node2, and supply command_submit and and command_terminate such that these commands can run on node1 on behalf of node2. I don't know if this would actually work.

[yonisidi]

we do have an api to submit batch jobs, could I write a controller plugin that interacts with the api instead of sshing? i am not sure though how lib or data dependencies would be transferred. apologies if this is in the docs already, I am still trying to find my footing in the package.

[wlandau]

The custom launch_worker() method you write in the plugin should give you that level of control.