Ploting POINT sf with geom_sf() is slow

[yutannihilation]

In my environment, it takes 10 seconds for plotting 1000 POINTs by geom_sf().
I feel this is a bit too long for exploring the data.

https://gist.github.com/yutannihilation/479cb5f254826915a7a3d36ff84b4b43

I suspect this is because draw_panel() applies sf_as_grob() on every row. Is it possible to do this more efficiently, e.g. st_combine() per group so that it plots one MULTIPOINT that contains 1000 points?

[clauswilke]

Could you check whether there is any relation to issue #2655?

[yutannihilation]

Thanks, I roughly think this is a different issue because I'm using Windows (sorry I forgot to write this). But, I will check further.

[yutannihilation]

Here's the result of profvis of the code that plots 1000 POINTs. This shows st_grob() takes most of the time.

[yutannihilation]

Here's a reprex. This shows st_combine() significantly save time.

library(ggplot2)
library(sf)
#> Linking to GEOS 3.6.1, GDAL 2.2.3, proj.4 4.9.3

set.seed(1)
n <- 1000

d <- data.frame(
  x = runif(n) * 360,
  y = runif(n) * 180
)

d_sf <- st_multipoint(as.matrix(d)) %>%
  st_sfc() %>% 
  st_cast("POINT")

system.time({
  p <- ggplot(d_sf) +
    geom_sf()
  print(p)
})

#>    user  system elapsed 
#>    7.36    0.10    7.71


system.time({
  d_sf_combined <- st_combine(d_sf)
  p <- ggplot(d_sf_combined) +
    geom_sf()
  print(p)
})

#>    user  system elapsed 
#>    0.27    0.05    0.31

Created on 2018-06-28 by the reprex package (v0.2.0).

Session info

devtools::session_info()
#> ─ Session info ──────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 3.5.0 (2018-04-23)
#>  os       Windows 10 x64              
#>  system   x86_64, mingw32             
#>  ui       RTerm                       
#>  language (EN)                        
#>  collate  Japanese_Japan.932          
#>  tz       Asia/Tokyo                  
#>  date     2018-06-28                  
#> 
#> ─ Packages ──────────────────────────────────────────────────────────────
#>  package     * version     date       source                            
#>  assertthat    0.2.0       2017-04-11 CRAN (R 3.5.0)                    
#>  backports     1.1.2       2017-12-13 CRAN (R 3.5.0)                    
#>  bindr         0.1.1       2018-03-13 CRAN (R 3.5.0)                    
#>  bindrcpp      0.2.2       2018-03-29 CRAN (R 3.5.0)                    
#>  callr         2.0.4       2018-05-15 CRAN (R 3.5.0)                    
#>  class         7.3-14      2015-08-30 CRAN (R 3.5.0)                    
#>  classInt      0.2-3       2018-04-16 CRAN (R 3.5.0)                    
#>  cli           1.0.0       2017-11-05 CRAN (R 3.5.0)                    
#>  clisymbols    1.2.0       2017-05-21 CRAN (R 3.5.0)                    
#>  colorspace    1.3-2       2016-12-14 CRAN (R 3.5.0)                    
#>  crayon        1.3.4       2017-09-16 CRAN (R 3.5.0)                    
#>  curl          3.2         2018-03-28 CRAN (R 3.5.0)                    
#>  DBI           1.0.0       2018-05-02 CRAN (R 3.5.0)                    
#>  debugme       1.1.0       2017-10-22 CRAN (R 3.5.0)                    
#>  desc          1.2.0       2018-05-01 CRAN (R 3.5.0)                    
#>  devtools      1.13.5.9000 2018-05-17 Github (r-lib/devtools@13ee56b)   
#>  digest        0.6.15      2018-01-28 CRAN (R 3.5.0)                    
#>  dplyr         0.7.5       2018-05-19 CRAN (R 3.5.0)                    
#>  e1071         1.6-8       2017-02-02 CRAN (R 3.5.0)                    
#>  evaluate      0.10.1      2017-06-24 CRAN (R 3.5.0)                    
#>  ggplot2     * 2.2.1.9000  2018-06-26 Github (tidyverse/ggplot2@348b26f)
#>  glue          1.2.0       2017-10-29 CRAN (R 3.5.0)                    
#>  gtable        0.2.0       2016-02-26 CRAN (R 3.5.0)                    
#>  htmltools     0.3.6       2017-04-28 CRAN (R 3.5.0)                    
#>  httr          1.3.1       2017-08-20 CRAN (R 3.5.0)                    
#>  knitr         1.20.2      2018-05-10 local                             
#>  lazyeval      0.2.1       2017-10-29 CRAN (R 3.5.0)                    
#>  magrittr      1.5         2014-11-22 CRAN (R 3.5.0)                    
#>  memoise       1.1.0       2018-06-13 Github (hadley/memoise@06d16ec)   
#>  mime          0.5         2016-07-07 CRAN (R 3.5.0)                    
#>  munsell       0.5.0       2018-06-12 CRAN (R 3.5.0)                    
#>  pillar        1.2.3       2018-05-25 CRAN (R 3.5.0)                    
#>  pkgbuild      1.0.0       2018-05-17 Github (r-lib/pkgbuild@0457039)   
#>  pkgconfig     2.0.1       2017-03-21 CRAN (R 3.5.0)                    
#>  pkgload       1.0.0       2018-05-17 Github (r-lib/pkgload@35efedd)    
#>  plyr          1.8.4       2016-06-08 CRAN (R 3.5.0)                    
#>  processx      3.1.0       2018-05-15 CRAN (R 3.5.0)                    
#>  purrr         0.2.5       2018-05-29 CRAN (R 3.5.0)                    
#>  R6            2.2.2       2017-06-17 CRAN (R 3.5.0)                    
#>  Rcpp          0.12.17     2018-05-18 CRAN (R 3.5.0)                    
#>  rlang         0.2.1       2018-05-30 CRAN (R 3.5.0)                    
#>  rmarkdown     1.10        2018-06-11 CRAN (R 3.5.0)                    
#>  rprojroot     1.3-2       2018-01-03 CRAN (R 3.5.0)                    
#>  scales        0.5.0.9000  2018-06-26 Github (hadley/scales@9e5e4d4)    
#>  sessioninfo   1.0.0       2017-06-21 CRAN (R 3.5.0)                    
#>  sf          * 0.6-3       2018-05-17 CRAN (R 3.5.0)                    
#>  spData        0.2.9.0     2018-06-17 CRAN (R 3.5.0)                    
#>  stringi       1.2.3       2018-06-12 CRAN (R 3.5.0)                    
#>  stringr       1.3.1       2018-05-10 CRAN (R 3.5.0)                    
#>  testthat      2.0.0       2017-12-13 CRAN (R 3.5.0)                    
#>  tibble        1.4.2       2018-01-22 CRAN (R 3.5.0)                    
#>  tidyselect    0.2.4       2018-02-26 CRAN (R 3.5.0)                    
#>  units         0.6-0       2018-06-09 CRAN (R 3.5.0)                    
#>  usethis       1.3.0       2018-02-24 CRAN (R 3.5.0)                    
#>  withr         2.1.2       2018-06-26 Github (jimhester/withr@fe56f20)  
#>  xml2          1.2.0       2018-01-24 CRAN (R 3.5.0)                    
#>  yaml          2.1.19      2018-05-01 CRAN (R 3.5.0)

[yutannihilation]

Here's a rough implementation for this. Does this make sense? If so, I will try to make the code ready to merge.

(Sorry, I just wanted to make a PR on my repo to show codes, but I mistakenly created here... 🙄 )

https://github.com/tidyverse/ggplot2/pull/2722/files

[jschoeley]

Furthermore ggplot(xy_points_dataframe) + geom_point() + coord_map() renders massively faster on my system compared to ggplot(point_sf) + geom_sf(). Dot-density maps of population counts are completely out of the question using geom_sf() on my system. They work wonderfully with geom_point() though...

# Draw a point at the centroid coordinates of each European region

library(tidyverse)
library(eurostat)
library(sf)

# using geom_sf
system.time({
  p1 <-
    eurostat_geodata_60 %>%
    filter(LEVL_CODE == 3) %>%
    st_centroid() %>%
    ggplot() + geom_sf() + coord_sf(crs = st_crs(3785))
  print(p1)
})

##   user  system elapsed 
## 10.634   0.179   6.710 

# using geom_point
system.time({
  p2 <-
    eurostat_geodata_60 %>%
    filter(LEVL_CODE == 3) %>%
    st_centroid() %>% st_coordinates() %>% as_data_frame() %>%
    ggplot() + geom_point(aes(x = X, y = Y)) + coord_map(projection = 'mercator')
  print(p2)
})

##   user  system elapsed 
##  0.889   0.004   0.896 

[thomasp85]

Personally I’m not sure if it is the goal of ggplot2 to optimise the input data. It might lead to unexpected results, e.g. in the stacking order of the different features...

If a fail-safe approach could be found it might be worth it but otherwise just documenting that multiple POINT will render slowly...

There’s no problem combining geom_sf and geom_point btw

[yutannihilation]

Basically I agree with @thomasp85, but I feel this is unacceptably slow and should be fixed.
I don't want to let ggplot2 be the tool that only those who know how to "optimize" the data can use.

In the case of #2655, it was not the ggplot2's fault and it seems there's nothing we can do. But, in this case, considering plot.sfc_POINT() and plot.sfc_MULTIPOINT() can plot this 10+ times faster, I think something is wrong with the current implementation.

That said, I don't think my PR is a good way to fix this. Maybe is it better to implement st_as_grob() methods for sfc classes first on sf's side...?

[clauswilke]

At some point, we may have to bite the bullet and see if we can speed up grid. It seems absurd that we cannot draw thousands of points as individual grobs without experiencing extreme speed penalties. The same applies to drawing many individual text labels. Otherwise we will have to keep making design choices in ggplot2 that are driven by limitations of grid rather than by what would be a clean design.

E.g., I find this comment on geom_label() quite frustrating:
https://ggplot2.tidyverse.org/reference/geom_text.html#geom-label
We shouldn't need two different geoms to draw text labels, one that is fast and one that can draw a colored background. It's not that complicated a problem.

[thomasp85]

I agree - as I’ve stated before I really hope to get some time to review the whole stack with performance in mind

My problem in combining POINT to multipoint is that it will result in all points being the same colour - also if a line or polygon were mixed in with the points, it would loose its ordering in the drawing stack

[yutannihilation]

Hmm, thanks. I didn't notice the point that we can speed up grid. It's reasonable to wait for it and avoid adding complex tweaks. OK, please close this issue anytime.

One thing, this is NOT true, if you are talking about combining POINTs in ggplot2:

combining POINT to multipoint is that it will result in all points being the same colour

If this is about combining them outside ggplot2, you are right. That's why I felt this should happen inside draw_panel() to map aesthetics like colour.... (About the other concerns, I think you are right.)

[yutannihilation]

Sorry, may I confirm one more thing? I wonder if this is really true.

It seems absurd that we cannot draw thousands of points as individual grobs without experiencing extreme speed penalties.

As I showed above (#2718 (comment)), most of the time is spent for sf_grob(), which is to build grobs, not to draw grobs. IIUC, improving grid can save time for draw.grid(), but this looks not the case to me.

[jschoeley]

I‘ve experienced the performance hit in the drawing, not the building stage.

[clauswilke]

I think many things are slow. If I look at the code for utils::modifyList() I want to cry. I think ggplot2 needs a clean and fast replacement of it for the purpose for which it is used. grid::gpar() also does tons of things that slow it down.

Considering that ultimately all gpar() does is create a list this is pretty awful:

microbenchmark::microbenchmark(
  grid::gpar(color = "blue"), list(color = "blue")
)
#> Unit: nanoseconds
#>                        expr   min    lq     mean  median      uq      max
#>  grid::gpar(color = "blue") 18715 19526 220562.0 20031.0 21431.5 19759145
#>        list(color = "blue")   143   161    308.2   185.5   217.0     8740
#>  neval
#>    100
#>    100

Created on 2018-06-28 by the reprex package (v0.2.0).