plot_env can cause duplicated serialization of ggplot data

[traversc]

Here is an example:

Case 1: saving a ggplot from global

library(ggplot2)
set.seed(1)
df <- data.frame(x = rnorm(1e6))
g <- ggplot(df, aes(x = x)) + geom_density()
saveRDS(g, "/tmp/temp.rds", compress=F)
file.info("/tmp/temp.rds")$size
[1] 8244327

Case 2: saving a ggplot from within a function

myplotfun <- function() {
  set.seed(1)
  df <- data.frame(x = rnorm(1e6))
  g <- ggplot(df, aes(x = x)) + geom_density()
  return(g)
}
g <- myplotfun()

saveRDS(g, "/tmp/temp.rds", compress=F)
file.info("/tmp/temp.rds")$size
[1] 24253766

In one case, I generate a ggplot from the global environment. In the second case, I generate the same ggplot but from within my custom plot function. I then save both these plots to disk and compare the size. The plot from within the function is 3x the size on disk.

This is caused by plot_env. If an environment is not a special case (e.g. global), R saves everything within the environment, which could include the original data.frame as well as the ggplot object itself (and therefore, everything in it).

[thomasp85]

I can’t really see any way to avoid this. Your plotting object may contain expressions to be evaluated in the plotting environment so it has to travel along with the plot object

[traversc]

It might not be easy, but could you parse the objects you need and include it in a list instead of an env?

I think this would be a large benefit, since many packages have custom plotting functions based on ggplot.

Edit:

Also, doesn't this cause an issue in the global env case? E.g.:

y <- 1:10
df <- data.frame(x = rnorm(10))
g <- ggplot(df, aes(x = x, y = y)) + geom_point()
saveRDS(g, "/tmp/temp.rds")

rm(list=ls())
g <- readRDS("/tmp/temp.rds")
g
Error in FUN(X[[i]], ...) : object 'y' not found

[yutannihilation]

So, are you asking us to provide an alternative serialization form than RDS?

[traversc]

Sorry, no I was just asking if there might be a better way instead of tracking the environment. Unfortunately, having an env in a list where the list is in the env is not a situation saveRDS handles properly.

If it's not possible, please feel free to close.

[yutannihilation]

IIUC, it's not possible. Many of the magical parts of ggplot2 (or even R itself) come from the mechanism of environments, so you need to save everything if you want the same result (even though the success is not always ensured).

If you are really sure your plot is not dependent on any environments, it might be an option to remove and restore plot_env by yourself. I'm not sure to what extent this works, though...

library(ggplot2)

myplotfun <- function() {
  set.seed(1)
  df <- data.frame(x = rnorm(1e6))
  g <- ggplot(df, aes(x = x)) + geom_density()
  return(g)
}
g <- myplotfun()

g$plot_env <- NULL
saveRDS(g, "/tmp/temp.rds")

# (restart R session)

library(ggplot2)

g <- readRDS("/tmp/temp.rds")
g$plot_env <- .GlobalEnv
g

[wlandau]

What about alternative serialization of ggplot objects like @yutannihilation mentioned in #3619 (comment)? I do not think it is worth trying to fight RDS or qs, but I feel as though an entirely different format may be worth pursuing.

[thomasp85]

This is really outside of the scope and interest of the core team. Others are free to pursue this if wanted.

The best way to save a ggplot (outside of rendering it) is to save the data along with the code. We do not ensure that ggplot objects are forward-compatible between versions, so saving the actual object may give a false sense of security

[combiz]

Just leaving this here to help anyone in a similar situation possibly. We have a scenario where we render a customized Rmarkdown report from a template via rmarkdown::render() by passing in a complex list of objects (e.g. lists, dfs, plots, etc.) saved immediately prior using saveRDS() or qs::qsave(). We had the issue that these rds/qs objects were often ballooning to 20Gb+ due to objects in the environment. In our case, saving the raw data and writing ggplot code within the report instead was not optimal (duplication). The solution was to selectively assign NULL to plot_env variables from problematic classes. In our case, 'SingleCellExperiment', 'ggplot', and ellipses were the common culprits. Our package runs this function on ggplot objects used in reports and the rds/qs files are appropriately sized, while being reproduced correctly in the rendered report (and still working nicely with plotly::ggplotly()): -

################################################################################
#' Remove specific classes from a ggplot plot_env environment
#'
#' Useful to remove large objects before writing to disk with qs or rds
#'
#' @family helper
#'
#' @importFrom purrr map_lgl
#'
#' @keywords internal
.clean_ggplot_plot_env <- function(p,
                                   drop_classes = c(
                                     "SingleCellExperiment",
                                     "ggplot"
                                   )) {
  if ("ggplot" %in% class(p)) {
    env_classes <- lapply(p$plot_env, class)
    drop_idx <- purrr::map_lgl(env_classes, ~ any(drop_classes %in% .))
    drop_names <- names(drop_idx[drop_idx])
    for (drop_name in drop_names) {
      p$plot_env[[eval(drop_name)]] <- NULL
    }
    p$plot_env$... <- NULL
  }
  return(p)
}

[clauswilke]

I think plot_env should be removed from the ggplot2 object/set to an empty environment. I've filed an issue: #3994

You can probably simplify your code to something like this:

library(ggplot2)

p <- ggplot(iris, aes(Sepal.Length, Sepal.Width, color = Species)) +
  geom_point() +
  facet_wrap(~Species)

p$plot_env <- rlang::new_environment()

p

^{Created on 2020-05-13 by the reprex package (v0.3.0)}