Memory leak when creating data.frames

[vspinu]

cpp_source the following snippet

#include "cpp11.hpp"
using namespace cpp11;

[[cpp11::register]]
data_frame c_memtest(int N=100000) {
  writable::doubles x1(N), x2(N), x3(N), x4(N);
  return writable::data_frame({
	  "x1"_nm = x1,
	  "x2"_nm = x2,
	  "x3"_nm = x3,
	  "x4"_nm = x4
	});
}

then evaluate the following till R blows out of memory:

while(T) {
  tt <- c_memtest(1e8)
  print(gc())
}
##             used   (Mb) gc trigger   (Mb)  max used (Mb)
## Ncells   2081367  111.2    4186895  223.7   2695081  144
## Vcells 403594716 3079.2  846406880 6457.6 803594699 6131
##             used   (Mb) gc trigger    (Mb)   max used   (Mb)
## Ncells   2082461  111.3    4186895   223.7    2695081  144.0
## Vcells 803597569 6131.0 1326410315 10119.8 1203597555 9182.8
##              used   (Mb) gc trigger    (Mb)   max used    (Mb)
## Ncells    2082473  111.3    4186895   223.7    2695081   144.0
## Vcells 1203597579 9182.8 2023772392 15440.2 1603597565 12234.5
##              used    (Mb) gc trigger    (Mb)   max used    (Mb)
## Ncells    2082485   111.3    4186895   223.7    2695081   144.0
## Vcells 1603597589 12234.5 2428606870 18528.8 2003597711 15286.3
##              used    (Mb) gc trigger    (Mb)   max used  (Mb)
## Ncells    2082497   111.3    4186895   223.7    2695081   144
## Vcells 2003597599 15286.3 2914408244 22235.2 2403597721 18338
##              used    (Mb) gc trigger    (Mb)   max used    (Mb)
## Ncells    2082509   111.3    4186895   223.7    2695081   144.0
## Vcells 2403597609 18338.0 3497369892 26682.9 2803597731 21389.8

> devtools::session_info()
─ Session info ──────────────────────────────────────────────────────────────────────────
 setting  value                       
 version  R version 4.0.0 (2020-04-24)
 os       Ubuntu 18.04.5 LTS          
 system   x86_64, linux-gnu           
 ui       X11                         
 language (EN)                        
 collate  en_US.UTF-8                 
 ctype    en_US.UTF-8                 
 tz       Europe/Berlin               
 date     2021-05-29                  

─ Packages ──────────────────────────────────────────────────────────────────────────────
 package     * version    date       lib source                       
 assertthat    0.2.1      2019-03-21 [1] CRAN (R 4.0.0)               
 brio          1.1.1.9000 2021-02-24 [1] Github (r-lib/brio@2cf72bb)  
 cachem        1.0.4      2021-02-13 [1] CRAN (R 4.0.0)               
 callr         3.5.1      2020-10-13 [1] CRAN (R 4.0.0)               
 cli           2.3.1      2021-02-23 [1] CRAN (R 4.0.0)               
 cpp11         0.2.7.9000 2021-05-29 [1] Github (r-lib/cpp11@d761dfc) 
 crayon        1.4.1.9000 2021-02-24 [1] Github (r-lib/crayon@965d1dc)
 debugme       1.1.0      2017-10-22 [1] CRAN (R 4.0.0)               
 decor         1.0.0      2020-06-30 [1] CRAN (R 4.0.0)               
 desc          1.2.0      2018-05-01 [1] CRAN (R 4.0.0)               
 devtools      2.3.2      2020-09-18 [1] CRAN (R 4.0.0)               
 ellipsis      0.3.1      2020-05-15 [1] CRAN (R 4.0.0)               
 fansi         0.4.2      2021-01-15 [1] CRAN (R 4.0.0)               
 fastmap       1.1.0      2021-01-25 [1] CRAN (R 4.0.0)               
 fs            1.5.0      2020-07-31 [1] CRAN (R 4.0.0)               
 glue          1.4.2      2020-08-27 [1] CRAN (R 4.0.0)               
 lifecycle     1.0.0      2021-02-15 [1] CRAN (R 4.0.0)               
 magrittr      2.0.1      2020-11-17 [1] CRAN (R 4.0.0)               
 memoise       2.0.0      2021-01-26 [1] CRAN (R 4.0.0)               
 pillar        1.5.0      2021-02-22 [1] CRAN (R 4.0.0)               
 pkgbuild      1.2.0      2020-12-15 [1] CRAN (R 4.0.0)               
 pkgconfig     2.0.3      2019-09-22 [1] CRAN (R 4.0.0)               
 pkgload       1.1.0      2020-05-29 [1] CRAN (R 4.0.0)               
 prettyunits   1.1.1      2020-01-24 [1] CRAN (R 4.0.0)               
 processx      3.4.5      2020-11-30 [1] CRAN (R 4.0.0)               
 ps            1.5.0      2020-12-05 [1] CRAN (R 4.0.0)               
 purrr         0.3.4      2020-04-17 [1] CRAN (R 4.0.0)               
 R6            2.5.0      2020-10-28 [1] CRAN (R 4.0.0)               
 remotes       2.2.0      2020-07-21 [1] CRAN (R 4.0.0)               
 rlang         0.4.10     2020-12-30 [1] CRAN (R 4.0.0)               
 rprojroot     2.0.2      2020-11-15 [1] CRAN (R 4.0.0)               
 sessioninfo   1.1.1      2018-11-05 [1] CRAN (R 4.0.0)               
 testthat      3.0.2      2021-02-14 [1] CRAN (R 4.0.0)               
 tibble        3.0.6      2021-01-29 [1] CRAN (R 4.0.0)               
 usethis       2.0.1      2021-02-10 [1] CRAN (R 4.0.0)               
 utf8          1.1.4      2018-05-24 [1] CRAN (R 4.0.0)               
 vctrs         0.3.6      2020-12-17 [1] CRAN (R 4.0.0)               
 withr         2.4.1      2021-01-26 [1] CRAN (R 4.0.0)               

[1] /home/vspinu/.lib/4.0.0
[2] /home/vspinu/bin/R-4.0.0-bin/library
> 

[jimhester]

Ok, should be fixed now, thank you for opening the issue and for supplying a reproducible example, it is a big help!

This was actually a fairly pernicious issue, but it should be fixed now. I will do a cpp11 release early next week to get this change out.

cpp11::cpp_source(code = '
#include "cpp11.hpp"
using namespace cpp11;

[[cpp11::register]]
data_frame c_memtest(int N=100000) {
  writable::doubles x1(N), x2(N), x3(N), x4(N);
  return writable::data_frame({
      "x1"_nm = x1,
      "x2"_nm = x2,
      "x3"_nm = x3,
      "x4"_nm = x4
    });
}
')

for(i in 1:10) {
  x <- c_memtest(1e6)
  print(gc())
}
#>           used (Mb) gc trigger  (Mb) limit (Mb) max used (Mb)
#> Ncells  748233   40    1284990  68.7         NA  1284990 68.7
#> Vcells 5370736   41   13338102 101.8      32768  9370719 71.5
#>           used (Mb) gc trigger  (Mb) limit (Mb) max used  (Mb)
#> Ncells  748769   40    1284990  68.7         NA  1284990  68.7
#> Vcells 5372358   41   18140074 138.4      32768 13372354 102.1
#>           used (Mb) gc trigger  (Mb) limit (Mb) max used  (Mb)
#> Ncells  748772   40    1284990  68.7         NA  1284990  68.7
#> Vcells 5372386   41   18140074 138.4      32768 13372491 102.1
#>           used (Mb) gc trigger  (Mb) limit (Mb) max used  (Mb)
#> Ncells  748774   40    1284990  68.7         NA  1284990  68.7
#> Vcells 5372410   41   18140074 138.4      32768 13372523 102.1
#>           used (Mb) gc trigger  (Mb) limit (Mb) max used  (Mb)
#> Ncells  748776   40    1284990  68.7         NA  1284990  68.7
#> Vcells 5372426   41   18140074 138.4      32768 13372563 102.1
#>           used (Mb) gc trigger  (Mb) limit (Mb) max used  (Mb)
#> Ncells  748778   40    1284990  68.7         NA  1284990  68.7
#> Vcells 5372442   41   18140074 138.4      32768 13372579 102.1
#>           used (Mb) gc trigger  (Mb) limit (Mb) max used  (Mb)
#> Ncells  748780   40    1284990  68.7         NA  1284990  68.7
#> Vcells 5372460   41   18140074 138.4      32768 13372598 102.1
#>           used (Mb) gc trigger  (Mb) limit (Mb) max used  (Mb)
#> Ncells  748782   40    1284990  68.7         NA  1284990  68.7
#> Vcells 5372479   41   18140074 138.4      32768 13372625 102.1
#>           used (Mb) gc trigger  (Mb) limit (Mb) max used  (Mb)
#> Ncells  748784   40    1284990  68.7         NA  1284990  68.7
#> Vcells 5372498   41   18140074 138.4      32768 13372653 102.1
#>           used (Mb) gc trigger  (Mb) limit (Mb) max used  (Mb)
#> Ncells  748786   40    1284990  68.7         NA  1284990  68.7
#> Vcells 5372517   41   18140074 138.4      32768 13372681 102.1

^{Created on 2021-06-04 by the reprex package (v2.0.0)}

[vspinu]

Confirmed, it's working as expected. I wonder, how did you manage to debug this and pinpoint the issue? Is there a way to print preserved list from R?

[jimhester]

It is not really possible to print from R, but you can use cpp11::preserved.print() to have cpp11 print it.

cpp11/inst/include/cpp11/protect.hpp

Lines 254 to 259 in 6b38737

	void print() {
	for (SEXP head = list_; head != R_NilValue; head = CDR(head)) {
	REprintf("%x CAR: %x CDR: %x TAG: %x\n", head, CAR(head), CDR(head), TAG(head));
	}
	REprintf("---\n");
	}

But it in this case I did not do that, it was more a case of familiarity of the code base and narrowing the cases where things could possibly be going wrong to find the issue.