future.batchtools Error: BatchtoolsExpiration: Future ('doFuture2-1') expired

[tmspvn]

Dear developers,

I am trying to run some code by using future.batchtools on a slurm HPC but I keep having the following error:

Error: BatchtoolsExpiration: Future ('doFuture2-1') expired (registry path /scratch/tpavan1/scripts_tommaso/test_batchtools/.future/20240710_190527-gmNN9y/doFuture2-1_1056043333).. The last few lines of the logged output:

However, the logged output changes while sometimes with a warming:

WARNING: While bind mounting '/users:/users': destination is already in the mount point list
 WARNING: While bind mounting '/tmp:/tmp': destination is already in the mount point list

Some other times nothing, and rarely it runs. I haven't been able to solve or figure out the reason why it's not running since it worked for a while and now it return the error above.

This is the test code i am running:

R is loadel via module load r-full/4.4.1 which loads a container with the R installation. To make R communicate with slurm I exported the following paths:

export SINGULARITY_BINDPATH="$SINGULARITY_BINDPATH,/etc/passwd,/var/run/munge,/usr/lib64/libmunge.so.2.0.0:/usr/lib64/libmunge.so.2,/run/slurm/conf/slurm.conf:/etc/slurm/slurm.conf,/usr/lib64/slurm,/usr/bin/sbatch,/usr/bin/squeue,/usr/bin/scancel"

After which I run the test code, which ran in my initial testing but the it started throwing errors.

library("future.batchtools")
options(future.debug = TRUE)
plan(batchtools_slurm, template="/slurm_singularity.tmpl") # not works
demo("mandelbrot", package = "future", ask = FALSE)

the template:

#!/bin/bash
#SBATCH --job-name=<%= job.name %>
#SBATCH --output=<%= log.file %>
#SBATCH --error=<%= log.file %>
#SBATCH --time=01:30:00
#SBATCH --mem=1G
#SBATCH --cpus-per-task=1
#SBATCH --nodes=1

module load r-full/4.4.1
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'

This is the session info:

R version 4.4.1 (2024-06-14)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 22.04.4 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.10.0 
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=fr_CH.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=fr_CH.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=fr_CH.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=fr_CH.UTF-8 LC_IDENTIFICATION=C       

time zone: Etc/UTC
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] future.batchtools_0.12.1 future_1.33.1            parallelly_1.36.0       

loaded via a namespace (and not attached):
 [1] crayon_1.5.2      vctrs_0.6.5       cli_3.6.2         rlang_1.1.3      
 [5] stringi_1.8.3     brew_1.0-10       data.table_1.15.0 batchtools_0.9.17
 [9] prettyunits_1.2.0 listenv_0.9.1     backports_1.4.1   hms_1.1.3        
[13] rappdirs_0.3.3    base64url_1.4     debugme_1.1.0     progress_1.2.3   
[17] lifecycle_1.0.4   compiler_4.4.1    codetools_0.2-19  fs_1.6.3         
[21] pkgconfig_2.0.3   digest_0.6.34     R6_2.5.1          parallel_4.4.1   
[25] checkmate_2.3.1   tools_4.4.1       withr_3.0.0       globals_0.16.2   

Do you have any idea of why it stopped working?

[HenrikBengtsson]

This might be problem specific to running inside a Linux Container (Singularity/Apptainer) - I'm not sure. Yeah, it could be a matter of binding the proper folders.

future.batchtools 0.20.0 is on CRAN as of 2025-08-25. It comes with lots of improvements (https://future.batchtools.futureverse.org/news/index.html). It should also give a bit more information when there are errors. Note also how the built-in templates are now more powerful, so you probably don't need your own custom one, e.g. https://future.batchtools.futureverse.org/reference/batchtools_slurm.html

Please retry with this version and see if it works or if there are more clues in what's going on.

[HenrikBengtsson]

Update on 2025-09-14. The development version of future.batchtools and its new built-in job script templates gained new skills that allows you to run R in Linux containers and set environment variables. Importantly, it also patches a bug in batchtools that may cause the "expiration" error. Install it as:

remotes::install_github("futureverse/future.batchtools", ref="develop")

See https://future.batchtools.futureverse.org/reference/batchtools_slurm.html for an example. See also futureverse/future.batchtools#100.

@tmspvn, looking at your example, try:

library(future)
plan(future.batchtools::batchtools_slurm, resources = list(
  time = "01:30:00", mem = "1G", nodes = 1, "cpus-per-task" = 1,
  modules = c("r-full/4.4.1")
))

f <- future(sessionInfo())
info <- value(f)
print(info)

If that doesn't work, you should get a more informative error message now.