Weird EMFILE error while open files limit is not reached

[bloodcarter]

• Version: v7.2.1
• Platform: Linux ip-172-31-16-252 4.9.43-17.39.amzn1.x86_64 Update README for help #1 SMP Fri Sep 15 23:39:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Subsystem:

I have a nodejs app constantly spawning and stopping child processes and doing fetch requests. After running for some time I am starting getting EMFILE errors:

 Error: getaddrinfo EMFILE www.googleapis.com:443
    at Object.exports._errnoException (util.js:1022:11)
    at errnoException (dns.js:33:15)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:76:26)

After that error immediatedly goes this one:

Error: spawn /home/ec2-user/workspace/memory/cognitive-task/forward-transformer/speech-parser/tomita-linux64 EMFILE
    at exports._errnoException (util.js:1022:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:193:32)
    at onErrorNT (internal/child_process.js:359:16)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickDomainCallback (internal/process/next_tick.js:122:9)

Command lsof | grep node shows something like this, i.e. max number of opened file descriptor is 80:

.....
node       3577 ec2-user   78u     unix 0xffff88007a594400       0t0 10332349 socket
node       3577 ec2-user   79u     unix 0xffff88007a597000       0t0 10332235 socket
node       3577 ec2-user   80u     unix 0xffff880004719c00       0t0 10332567 socket
......

If I do lsof | grep node | wc -l I yelds no more than 250 in total.

Output of ulimit -a is:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7936
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 7936
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

It shows that I have limit of 4096 file descriptors and I run simultaneously maybe 10.
I have no idea what is going on and how to fix it. I am pretty sure that no one process stucks or something.

[gireeshpunathil]

can you also occasionally do ps -ef | grep node | wc -l to make sure that the app is not leaking child processes?

[bloodcarter]

Yes, I did. It is not leaking. The command outputs only 12 processes I really use.

[gireeshpunathil]

@bloodcarter - sorry for the delay. Is it possible to share the leaking code? I am try debugging.