Bulk loader uses too much memory

[codexnull]

The bulk loader's memory usage can grow beyond the size of physical RAM and cause swap space to be used. As the image below shows, memory usage grows monotonically during the map phase, is flat during the reduce phase, and is all finally released at exit.

The system does not appear to thrash, though, which suggests the memory is just being held by the process without it actively being accessed. Still, a big enough load may cause it to grow greater than RAM + swap and cause system instability or crashes.

Note the system was idle other than for dgraph. I did not test what would happen if concurrent processes had significant memory demands as well.

[shekarm]

Hi,

Thank you for the post. We have made some improvements in our software (recently) which will likely exhibit different behavior -- in other words, much less memory usage. Keep you posted on developments. Meanwhile, if you haven't already done so, would you be able to share the characteristics of your data set that you are using?

Thanks

[ashish-goswami]

I think this issue can be closed after we fix #5361

[minhaj-shakeel]

Github issues have been deprecated.
This issue has been moved to discuss. You can follow the conversation there and also subscribe to updates by changing your notification preferences.