Limiting the number of workers.

[joshualerickson]

Hey future community. A bit of a random question, but recently I was contacted by a package maintainer to lessen the number of parallelized server requests within a package I'm working on due to possibly crashing the server when requests are too high. They suggested hard coding it into the functions so that 10 or less workers/cores would be used. My question is two-fold; 1). has anyone ever done this with the future framework, 2.) how could this be done while still following the practices recommended by this vignette? Thanks!

[HenrikBengtsson]

I'm not sure I'm following. You have "future" code inside a package of yours, and there's another package, whose maintainer asks you to reconfigure your package. Can you share more details, e.g. your package, or at least the piece of code that is relevant?

Also, as a wild guess of what you might be doing: The motto and design philosophy is that the developer choose what code to parallelize, whereas the end-user choose how and where to parallelize. In other words, it's always best to leave it to the end-user to set the plan(), because as a developer, you can never know what compute resources the end-user has. See also ?plan. Are you setting the plan() inside your package?

[joshualerickson]

For sure, here is the package and a little snippet of code below. I use the furrr::future_map() to setup the code and then it's up to the user to decide how and where. I really like that framework! But, the code that I've setup uses an API call and the maintainer of the server is worried that this might be causing the server to crash... They suggested having a limit on how many workers/cores the user can use. I'm torn because I like being able to call plan(multisession) once at the the beginning and not have to worry about what the user might need/want. However, I also don't want to have a package that potentially is crashing a server. Let me know if that makes sense? Thanks again.

#need to call future::plan()
plan(multisession(workers = availableCores()-1))

#running on 11 cores

system.time({
pnw_dv <- suppressMessages(ww_dvUSGS(huc17_sites$site_no,
                    parameter_cd = '00060',
                    wy_month = 10,
                    parallel = TRUE))
})

## under the hood ww_dvUSGS() is calling furrr::future_map() when parallel = TRUE

[HenrikBengtsson]

I see. So, the problem is that you hit their servers too frequently. It sounds like you're looking for a way to throttle how many requests you do within some period of time, say, requests per minute. There's nothing specific particular to parallel processing with your problem. You could imagine hitting a server too often also with vanilla R sequential code. So, I would attack the problem as such. Of course, you can limit the amount of traffic you produce by limiting the number of parallel workers hitting the server, but that's a bit like using the wrong tool for what should be done.

Instead, I think you want to inject some Sys.sleep(delay) between each of your calls, and the amount of delay (seconds), needs to be a function of the number of parallel workers, which you can get by nbrOfWorkers(). Something like:

## max 30 requests per minute
max_freq_per_min <- 30

## delay in seconds between requests
delay <- 60/max_freq_per_min * nbrOfWorkers()

When running sequentially, you'll get delay = 2.0 seconds. With two parallel workers, you'll get delay = 4.0 seconds, and so on. Then, use Sys.sleep(delay) in your parallel loop/lapply/map call.

[joshualerickson]

Yes! I was thinking of the wrong tool. So by injecting a delay, the background processes will be staggered enough so that the server is not hit too frequently but still advantageous to use parallel. If coded appropriately, then it shouldn’t matter whether 50 or 2 workers are be used? Thank you for your time and insight!