Formula defined outside %dopar% cannot get variables in Global environment

[PierreMasselot]

Hi,

I ran into an error while performing analyses in parallel, with a formula defined before the %dopar% loop that needs to get variables in the global environment.
The following code throws an error:

x <- rnorm(100)
y <- 5 + 2 * x + rnorm(100, 0, .2)
form <- y ~ x

cl <- makeCluster(2)
registerDoParallel(cl)
foreach(i = 1:2, .packages = "mixmeta") %dopar% {
  model <- lm(form)
}
stopCluster(cl)

On the other hand, either putting the variables into a data.frame or defining the formula within the %dopar% loop work fine.

x <- rnorm(100)
y <- 5 + 2 * x + rnorm(100, 0, .2)
df <- data.frame(y = y, x = x)
form <- y ~ x

cl <- makeCluster(2)
registerDoParallel(cl)
foreach(i = 1:2) %dopar% {
  model <- lm(form, data = df)
}
stopCluster(cl)

x <- rnorm(100)
y <- 5 + 2 * x + rnorm(100, 0, .2)

cl <- makeCluster(2)
registerDoParallel(cl)
foreach(i = 1:2) %dopar% {
  model <- lm(y ~ x)
}
stopCluster(cl)

This is not a crippling issue as it is easy to find a workaround, but I thought you might want to know.

Thanks!

[HenrikBengtsson]

You need to guide %dopar% about which variables are needed to resolve lm(form), which you can do by:

res <- foreach(i = 1:2, .export = c("y", "x"), .packages = "mixmeta") %dopar% {
  model <- lm(form)
}

That'll do it.

[PierreMasselot]

Thanks, but unless I am missing something, I still have the same error in this case.

Here is the full code.

library(doParallel)

x <- rnorm(100)
y <- 5 + 2 * x + rnorm(100, 0, .2)
form <- y ~ x

cl <- makeCluster(2)
registerDoParallel(cl)
res <- foreach(i = 1:2, .export = c("y", "x")) %dopar% {
  model <- lm(form)
}
stopCluster(cl)

It seems to have something to do with the environment attached to the formula.

[HenrikBengtsson]

Thanks, but unless I am missing something, I still have the same error in this case.

Sorry, my bad - yes, I can reproduce this with doParallel.

If you're okay with moving away from doParallel, then it will work if you use doFuture, which parallelizes via futureverse. Here's a "one-to-one" update to your existing code example:

library(doFuture)
library(parallel)

x <- rnorm(100)
y <- 5 + 2 * x + rnorm(100, 0, .2)
form <- y ~ x

cl <- makeCluster(2)
plan(cluster, workers = cl)
registerDoFuture()

res <- foreach(i = 1:2, .export = c("y", "x")) %dopar% {
  model <- lm(form)
}

stopCluster(cl)

which gives:

> res
[[1]]

Call:
lm(formula = form)

Coefficients:
(Intercept)            x  
      5.023        1.982  


[[2]]

Call:
lm(formula = form)

Coefficients:
(Intercept)            x  
      5.023        1.982  

Since, plan(multisession, workers = 2) is effectively the same as cl <- makeCluster(2) + plan(cluster, workers = cl), but is also better for other reasons, the next simplification is:

library(doFuture)

x <- rnorm(100)
y <- 5 + 2 * x + rnorm(100, 0, .2)
form <- y ~ x

plan(multisession, workers = 2)
registerDoFuture()

res <- foreach(i = 1:2, .export = c("y", "x")) %dopar% {
  model <- lm(form)
}

plan(sequential) # stop parallel cluster

Instead of .export = c("y", "x"), you could add "dummy" place-holders of those two variables inside the foreach expression, to guide foreach and futureverse that those are also needed, e.g.

res <- foreach(i = 1:2) %dopar% {
  ## Hint that 'x' and 'y' are also needed
  x; y

  model <- lm(form)
}

I prefer this style since it removes some clutter from the foreach() call.

Finally, I advocate for %dofuture% instead of %dopar% for reasons explained in my 2023-06-26 blog post '%dofuture% - a Better foreach() Parallelization Operator than %dopar%' (https://www.jottr.org/2023/06/26/dofuture/). So, in the end of the day I recommend to use:

library(doFuture)
plan(multisession, workers = 2)

x <- rnorm(100)
y <- 5 + 2 * x + rnorm(100, 0, .2)
form <- y ~ x

res <- foreach(i = 1:2) %dofuture% {
  ## Hint that 'x' and 'y' are also needed
  x; y

  model <- lm(form)
}

plan(sequential) # stop parallel cluster

Note how there's no registerDoNnn(), which is one of the advantages.

PS. It is not unlikely that a future version of futureverse will be able to automatically infer that x and y is part of formula form and you no longer will have to declare them explicitly. That's at least my hope to be able to support.

[PierreMasselot]

Ha, I was not aware of doFuture, thanks for flagging that up. I will have a look.