Sysreqs calls are too many if lots of deps

[muschellij2]

It seems as though you're using sysreqs to get the system requirements for a container and it seems you always get the list of packages, either from the session info, description, etc.

Once you have the list of packages, you seem to loop through each package. This is not required as sysreqs can handle a character vector of packages, such as https://sysreqs.r-hub.io/pkg/containerit,igraph,xml2,magick. The issue here is that the looping, for a large number of dependencies essentially DDoS's the API and then you get some 400's. I propose to stop the looping in .find_system_dependencies.

library(httr)
library(sysreqs)
platform = "linux-x86_64-debian-gcc"

all_deps <- sysreqs:::get_cran_deps(c("containerit", "remotes", 
                                      "dplyr", "httr", "xml2", 
                                      # bad package
                                      "casdfsdf2343535"))
dep_sysreqs <- sysreqs:::get_cran_sysreqs(
  all_deps, 
  platform = platform)
dep_sysreqs
#> [1] "libssl-dev"           "libxml2-dev"          "libcurl4-openssl-dev"
#> [4] "git-core"

package = all_deps
package <- paste(package, collapse = ",")
url <- sysreqs:::make_url(sysreqs:::sysreqs_cran_url, package = package, platform = platform)
print(url)
#> [1] "https://sysreqs.r-hub.io/pkg/containerit,remotes,dplyr,httr,xml2,casdfsdf2343535,ellipsis,generics,glue,lifecycle,magrittr,R6,rlang,tibble,tidyselect,vctrs,pillar,cli,crayon,fansi,utf8,pkgconfig,purrr,curl,jsonlite,mime,openssl,askpass,sys/linux-x86_64-debian-gcc"
unlist(sysreqs:::download_json(url))
#> [1] "libssl-dev"           "libxml2-dev"          "libcurl4-openssl-dev"
#> [4] "git-core"

^{Created on 2021-11-19 by the reprex package (v2.0.0)}

Session info

sessioninfo::session_info()
#> ─ Session info  ──────────────────────────────────────────────────────────────
#>  hash: woman health worker: medium skin tone, white small square, clapping hands: medium skin tone
#> 
#>  setting  value
#>  version  R version 4.1.0 (2021-05-18)
#>  os       Debian GNU/Linux 10 (buster)
#>  system   x86_64, linux-gnu
#>  ui       X11
#>  language (EN)
#>  collate  C.UTF-8
#>  ctype    C.UTF-8
#>  tz       Etc/UTC
#>  date     2021-11-19
#>  pandoc   2.14.0.2 @ /opt/conda/bin/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version    date (UTC) lib source
#>  cli           3.1.0.9000 2021-10-29 [1] Github (r-lib/cli@1af3d91)
#>  crayon        1.4.2      2021-10-29 [1] CRAN (R 4.1.0)
#>  debugme       1.1.0      2017-10-22 [1] CRAN (R 4.1.0)
#>  desc          1.4.0      2021-09-28 [1] CRAN (R 4.1.0)
#>  digest        0.6.28     2021-09-23 [1] CRAN (R 4.1.0)
#>  evaluate      0.14       2019-05-28 [2] CRAN (R 4.1.0)
#>  fastmap       1.1.0      2021-01-25 [2] CRAN (R 4.1.0)
#>  fs            1.5.0      2020-07-31 [2] CRAN (R 4.1.0)
#>  glue          1.5.0      2021-11-07 [1] CRAN (R 4.1.0)
#>  highr         0.9        2021-04-16 [2] CRAN (R 4.1.0)
#>  htmltools     0.5.2      2021-08-25 [1] CRAN (R 4.1.0)
#>  httr        * 1.4.2.9000 2021-11-01 [1] Github (r-lib/httr@21ff69f)
#>  knitr         1.36       2021-09-29 [1] CRAN (R 4.1.0)
#>  magrittr      2.0.1      2020-11-17 [2] CRAN (R 4.1.0)
#>  R6            2.5.1      2021-08-19 [1] CRAN (R 4.1.0)
#>  reprex        2.0.0      2021-04-02 [2] CRAN (R 4.1.0)
#>  rlang         0.4.12     2021-10-18 [1] CRAN (R 4.1.0)
#>  rmarkdown     2.9        2021-06-15 [2] CRAN (R 4.1.0)
#>  rprojroot     2.0.2      2020-11-15 [2] CRAN (R 4.1.0)
#>  rstudioapi    0.13       2020-11-12 [2] CRAN (R 4.1.0)
#>  sessioninfo   1.1.1.9000 2021-10-22 [1] Github (muschellij2/sessioninfo@a4a39f1)
#>  stringi       1.7.5      2021-10-04 [1] CRAN (R 4.1.0)
#>  stringr       1.4.0      2019-02-10 [2] CRAN (R 4.1.0)
#>  sysreqs     * 1.0.0.9000 2021-10-08 [1] Github (r-hub/sysreqs@f068afa)
#>  withr         2.4.2      2021-04-18 [2] CRAN (R 4.1.0)
#>  xfun          0.27       2021-10-18 [1] CRAN (R 4.1.0)
#>  yaml          2.2.1      2020-02-01 [2] CRAN (R 4.1.0)
#> 
#>  [1] /home/jupyter/.R/library
#>  [2] /usr/local/lib/R/site-library
#>  [3] /usr/lib/R/site-library
#>  [4] /usr/lib/R/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

[muschellij2]

Also, now that I look into it more, it seems as though .find_by_sysreqs_pkg calls sysreqs
directly:
https://github.com/o2r-project/containerit/blob/master/R/package-installation-methods.R#L273
and
.find_by_sysreqs_api is calling the API:
https://github.com/o2r-project/containerit/blob/master/R/package-installation-methods.R#L302.

The issue is that sysreqs uses get_cran_deps https://github.com/r-hub/sysreqs/blob/master/R/sysreqs.R#L54, which calls make_url
https://github.com/r-hub/sysreqs/blob/master/R/sysreqs.R#L16, which calls the API command.

So, both functions are actually calling the API and so I think overall containerit should simply call sysreqs::sysreqs rather than the 2 functions. If I'm missing something, then I'd be interested to see why the 2 methods were created.

[muschellij2]

Looks like RSPM now has sysreqs in there: https://github.com/tidymodels/vetiver-r/blob/main/R/write-docker.R#L91