OOM when importing large directories

[achingbrain]

This call to get-folder-size can cause OOMs on large directories as the get-folder-size module keeps a list of all the files it's seen to make sure it doesn't double-count symlinked files (for example).

We only calculate the directory size to show a progress indicator - I propose we switch to an indeterminate indicator if the number of files being imported is unreasonable (say, 10k).

What do you think?

[alanshaw]

What if we switch to files added / total files for > 10K file adds? I'm assuming you're going to need to count the total files to determine if you're over 10K or not...this would give the user at least some indication of progress.

[achingbrain]

This is a little harder than it seems - the way all these file counter modules seem to work is to call fs.readdir which returns a list of files. If the directory is too big, it'll explode.

There's an issue on the node.js repo about an iterative fs.readdir but it needs support added to libuv first, which looks like is happening, but slowly.

[alanshaw]

I'm hoping that #1777 might help a little - if you call ipfs add --progress=false then get-folder-size won't be used anymore.

[achingbrain]

js-ipfs is being deprecated in favor of Helia. You can #4336 and read the migration guide.

@helia/unixfs doesn't notify on progress in the same way, the user can keep track of the number of files/folders imported if they wish to show feedback.