Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

event emitter memory leak on flush #3529

Closed
usernameisalreadytaken2014 opened this issue Oct 26, 2015 · 7 comments
Closed

event emitter memory leak on flush #3529

usernameisalreadytaken2014 opened this issue Oct 26, 2015 · 7 comments
Labels
events Issues and PRs related to the events subsystem / EventEmitter. zlib Issues and PRs related to the zlib subsystem.

Comments

@usernameisalreadytaken2014

I'm using SSE streams to push JSON events to a web browser.

There is a res.flush() after every bundle of SSE data, to make sure the client gets data.

(Every 15 seconds there's also a keep-alive function that sends a ": keep-alive\n" comment over the channel, along with a res.flush(), to keep stateful firewalls from closing the TCP connection while the browser window showing graphs etc. is open.)

Also using require('compression'), since the SSE streams are usually heavy with data in the beginning before then slowing down.

For some reason, res.flush() causes an error from node:

(node) warning: possible EventEmitter memory leak detected. 11 drain listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
    at Gzip.addListener (events.js:239:17)
    at Gzip.Readable.on (_stream_readable.js:665:33)
    at Gzip.once (events.js:265:8)
    at Gzip.Zlib.flush (zlib.js:448:10)
    at ServerResponse.flush (compression/index.js:70:16)
    at null._repeat (sample.js:2:8)
    at wrapper [as _onTimeout] (timers.js:264:19)
    at Timer.listOnTimeout (timers.js:92:15)

I couldn't find response.flush() documented in the v4.2.1 NodeJS documentation, so I might be calling it incorrectly?

As far as I could read from the compression module documentation, I'm using it correctly though..

According to the folks over at compression, this is not a bug in the compression library, but in nodejs:
expressjs/compression#58 (comment)

What is your opinion?

@usernameisalreadytaken2014
Copy link
Author

By the way, I'm unsure if flush() just inserts a marker/barrier in the stream, so that whatever stream reader that is chained to the one I'm writing to will see the marker and know that it's a good time to flush when it hits the marker? I think I've been assuming that write() and flush() cooperate approximately in such a way.

But maybe it's more intricate, and flush() actually causes some processing to happen, and I have to make write() and flush() cooperate manually, for example by holding off further write()s until the flush() has returned by executing a callback?

If anyone knows...

@mscdex mscdex added events Issues and PRs related to the events subsystem / EventEmitter. zlib Issues and PRs related to the zlib subsystem. labels Oct 26, 2015
@Fishrock123
Copy link
Contributor

@dougwilson could you detail the core bug a bit more?

@jonathanong
Copy link
Contributor

you're probably just calling flush too much. each flush call waits for a drain, and a drain probably never happened.

any reason you don't just use zlib.Z_SYNC_FLUSH? https://nodejs.org/api/zlib.html#zlib_constants

an optimization in node could maybe be knowing when a flush is in progress so it doesn't continuously add more listeners

@dougwilson
Copy link
Member

The bug is that every call to .flush will add an event listener, even if no callback was given, which makes the event listener pointless.

The history and the fix can be found in nodejs/node-v0.x-archive#25679

@usernameisalreadytaken2014
Copy link
Author

you're probably just calling flush too much.

Hmm, I'm calling flush() the minimum necessary amount of times, as far as I can tell...

(Browser requests data from a starting point in time; server sends all data from that point in time and up to the most recent, after which it will flush() once; then it sends events as they occur in the future, flushing once after each event.)

an optimization in node could maybe be knowing when a
flush is in progress so it doesn't continuously add more listeners

Could be. Are the listeners necessary?

any reason you don't just use zlib.Z_SYNC_FLUSH?

Is this question meant for the compress module developers?

@dougwilson
Copy link
Member

compression will not be changing the flush mode, and it has nothing to do with this issue here (changing the flush mode doesn't have any effect on the event listener leak). Details of the difference between the two flush mode was summed up at expressjs/compression#3 (comment)

@MylesBorins
Copy link
Contributor

Fix has landed on master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
events Issues and PRs related to the events subsystem / EventEmitter. zlib Issues and PRs related to the zlib subsystem.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants