Hint about memory usage in README

[anatolsommer]

It would be great if there was a hint that constantly calling pusage.stat (or fs.readFile/createReadStream in general) is filling up memory pretty fast if you don't call gc manually every few seconds. This might save a lot of people a lot of time.

[soyuka]

In fact we keep a history to get more accurate data on linux. To clean up this you have to use the unmonitor function. I don't think that fs.readFile is the issue here.

[anatolsommer]

It is. I spent more than 5 hours in tracking down that problem yesterday.

var fs=require('fs'), noop=function() {};

(function readFile() {
  for (var i=0; i<500; i++) {
    fs.readFile(__filename, noop);
  }
  //gc();
  setTimeout(readFile, 50);
})();

Result: 60MB RSS within a few seconds - if you uncomment gc() memory usage constantly stays below 20MB RSS. Tested on different VMs and linux distributions, surprisingly there seems to be no problem on windows.

[soyuka]

50ms is very low. I can improve that and will do, thanks.

[anatolsommer]

I started to inspect the problem, when my script calling pusage.stat every 1000ms used 80MB RSS after a few hours, so I can confirm this also affects real life situations. Thank you too.

[soyuka]

So, I tried to keep the fd cursor in memory to avoid (re)opening the file every time but it does not make any difference and fs.read() still uses memory and the gc() behavior is the same (commit).

May I ask what node version do you use? I launched my last test on v5.5 without improvements :|

[anatolsommer]

I used v0.12, v4.2.6 and v5.5 in my tests and saw no real difference between these versions.

[soyuka]

Thanks, I guess I'll just add a warning to the readme as I can't find any fix :|. Do you think the memory consumption is worth opening an issue on nodejs/node? Taking the fact that using gc() makes it behave normally, I'm wondering.

[anatolsommer]

Someone did report the bug, but they closed it since gc() solves the problem and it's not a real memory leak (at some point memory stops filling up). Sorry, can't find the link right now...

This comment is confusing since it uses "leaking". As @anatolsommer stated above, this is not a memory leak, which wouldn't be corrected by calling gc().

[retrohacker]

Is this a leak? Is the disclaimer necessary?

Won't Node.js automatically run the gc when it needs to? Running the gc() call every 100ms is extremely expensive.

[soyuka]

"leak" is a bit strong indeed, changed.