ENFILE when daemon is on

[daviddias]

I started seeing a ENFILE error when adding large directories with the daemon on. This is we are trying to open all the files to then send them through js-ipfs-api -- https://github.com/ipfs/js-ipfs/blob/master/src/cli/commands/files/add.js --.

Note that I have my ulimit already increased by default and not using Mac OS X default.

» ulimit -n
1024

This can be reproduced by simply adding the node_modules folder from js-ipfs.

» jsipfs daemon
Initializing daemon...
Swarm listening on /ip4/127.0.0.1/tcp/4003/ws/ipfs/Qmd1qCCBvyAVQKKB9Pkq5xhcwXSBZCiyrYJXNiYR6aV2nk
Swarm listening on /ip4/127.0.0.1/tcp/4002/ipfs/Qmd1qCCBvyAVQKKB9Pkq5xhcwXSBZCiyrYJXNiYR6aV2nk
Swarm listening on /ip4/172.20.6.141/tcp/4002/ipfs/Qmd1qCCBvyAVQKKB9Pkq5xhcwXSBZCiyrYJXNiYR6aV2nk
API is listening on: /ip4/127.0.0.1/tcp/5002
Gateway (readonly) is listening on: /ip4/127.0.0.1/tcp/9090
Daemon is ready

» jsipfs add -r node_modules
events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: ENFILE: file table overflow, open 'node_modules/fbjs/lib/TokenizeUtil.js.flow'
    at Error (native)

[grassias]

I can confirm this issue. The problem is that, in my machine, when I try to add a small directory (e.g. img folder from js-ipfs) with the daemon on, I get another error:

>> jsipfs add -r img/
/usr/local/lib/node_modules/ipfs/src/cli/commands/files/add.js:131
        throw err
        ^

Error: [object Object]
    at parseError (/usr/local/lib/node_modules/ipfs/node_modules/ipfs-api/src/request-api.js:16:17)
    at ClientRequest.<anonymous> (/usr/local/lib/node_modules/ipfs/node_modules/ipfs-api/src/request-api.js:38:14)
    at ClientRequest.g (events.js:291:16)
    at emitOne (events.js:96:13)
    at ClientRequest.emit (events.js:188:7)
    at HTTPParser.parserOnIncomingClient (_http_client.js:473:21)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:99:23)
    at Socket.socketOnData (_http_client.js:362:20)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)

Can someone reproduce it as well? Or there is something wrong with my configuration of js-ipfs? I am noting that my ulimit is 256 (default in macOS Sierra).

[alanshaw]

I haven't checked, but I think the problem is that the builder is trying to put ALL files into the store at once.

We can probably fix this by using parallelLimit and setting the limit to something sensible.

The ulimit on MacOS seems to be the strictest (256) so limit to 64?

[daviddias]

This issue only happens when the daemon is on and on the side of the CLI, when the daemon is off (meaning directly to code-> builder) it works fine.

The real issue is coming from https://github.com/ipfs/js-ipfs/blob/master/src/cli/commands/files/add.js trying to buffer everything to create the multipart message instead of streaming it.