fread: memory not being freed

[patrickhowerter]

I have found that reading data into a data.table will cause a memory leak. This seems to happen whether I use fread or read_fst(from the fst package). I suspect that this could even be an R or OS issue, however, I have not found any cases of this on Stack Overflow...

Steps to reproduce:

1. Download the sample file here:

download.file("https://mdodemodiag770.blob.core.windows.net/testcontainer/test.csv", "test.csv")

2. Create an R script as the following:

tst <- data.table::fread("tst.csv");
rm(tst);
gc()

3. Execute the R script and check the results on your OS. I am using Ubuntu 18.04.1. I see the following results of my R process when I run the top command (2.6 gbs in Virtual memory);

  PID   USER        PR VIRT         RES        SHR   S   %CUP %MEM TIME + COMMAND
15827 patrick+  20   0  2.625g 1.269g 0.013g S   0.0  1.2   0:12.93 rsession
 

--If you want to run in it with valgrind, you can use the following:
4. Run the following valgrind command (change the source script to the location of your R script from step 2):

R -d "valgrind --show-leak-kinds=all" -e "source('/home/patrick.howerter/valgrind_test.R')"
 

Here is the end of the valgrind output.
5.

==533==
==533== HEAP SUMMARY:
==533==     in use at exit: 134,671,721 bytes in 19,047 blocks
==533==   total heap usage: 42,608 allocs, 23,561 frees, 1,113,301,337 bytes allocated
==533==
==533== LEAK SUMMARY:
==533==    definitely lost: 0 bytes in 0 blocks
==533==    indirectly lost: 0 bytes in 0 blocks
==533==      possibly lost: 4,320 bytes in 15 blocks
==533==    still reachable: 134,667,401
 
bytes in 19,032 blocks
==533==         suppressed: 0 bytes in 0 blocks
==533== Rerun with --leak-check=full to see details of leaked memory
==533==
==533== For counts of detected and suppressed errors, rerun with: -v
==533== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
 

sessionInfo()

R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.1 LTS

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8   
 [6] LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] bit_1.1-14        compiler_3.5.2    tools_3.5.2       yaml_2.2.0        bit64_0.9-7       data.table_1.12.0

``

 
 

[patrickhowerter]

For a comparison, I tried the readr:read_csv function. This seems to free up most of the memory.

read <- readr::read_csv("/mnt/batch/tasks/shared/fileshare/mdo_data/downloads/TREstSum2000");
rm(read);
gc()

top command output:

PID   USER          PR   NI               VIRT    RES        SHR   S   %CUP %MEM TIME + COMMAND
10512 patrick+  20   0  465.4m 139.0m  25.5m S   0.0  0.1   0:36.46 rsession

If you run the read.csv command from base R, the same problem as fread.

#3719842      33
read <- read.csv("/mnt/batch/tasks/shared/fileshare/mdo_data/downloads/TREstSum2000");
rm(read);
gc()

top command results:

PID   USER          PR   NI               VIRT    RES        SHR   S   %CUP %MEM TIME + COMMAND
12169 patrick+  20   0 1879.1m 1.518g  24.6m S   0.3  1.4   1:44.99 rsession

[roysh913]

Just came across this post while solving memory leakage issue. One of the problems is still with fread.

require(data.table)

for(i in 1:10) {
  sampleDt <- fread("sample-data.csv", header = TRUE, stringsAsFactors = FALSE)
  rm(sampleDt)
  gc()
  print(pryr::mem_used())
}
# 54.3 MB
# 61.4 MB
# 68.3 MB
# 75.2 MB
# 82.2 MB
# 89.1 MB
# 96 MB
# 103 MB
# 110 MB
# 117 MB

Using read.csv has no leakage.

for(i in 1:10) {
  sampleDt <- read.csv("sample-data.csv", header = TRUE, stringsAsFactors = FALSE)
  rm(sampleDt)
  gc()
  print(pryr::mem_used())
}
# 47.7 MB
# 47.8 MB
# 47.8 MB
# 47.8 MB
# 47.8 MB
# 47.8 MB
# 47.8 MB
# 47.8 MB
# 47.8 MB
# 47.8 MB

sessionInfo()

R version 3.6.0 (2019-04-26)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 17763)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.12.2

loaded via a namespace (and not attached):
[1] compiler_3.6.0   pryr_0.1.4       magrittr_1.5     tools_3.6.0      Rcpp_1.0.1       stringi_1.4.3    codetools_0.2-16 stringr_1.4.0

[MichaelChirico]

Confirming reproducibly on Mac:

library(data.table)
tmp = tempfile()
iris = as.data.table(iris)
DT = rbindlist(replicate(1e5, iris, simplify = FALSE))
fwrite(DT, tmp)

for(i in 1:10) {
  sampleDt <- fread(tmp, header = TRUE, stringsAsFactors = FALSE)
  rm(sampleDt)
  gc()
  print(pryr::mem_used())
}
736 MB
875 MB
1.01 GB
1.15 GB
1.29 GB
1.43 GB
1.57 GB
1.71 GB
1.85 GB
1.98 GB

[realalexgalenko]

Finishing my 6 hour memory leak search here. Sadly, same issue here with fread. Running MacOS with 1.12.8 data.table.

My memory usage progression looks like

33 MB
283 MB
533 MB
783 MB
1.03 GB
1.28 GB
1.53 GB
1.78 GB
2.03 GB
2.28 GB