Performance issue in socket.io 1.3.6

[darrachequesne]

Hi! While benching socket.io for #2239 with https://github.com/darrachequesne/siobench (forked from https://github.com/mixu/siobench), I noticed a quite important decrease in performance between version 1.3.5 and 1.3.6. The memory footprint does increase a lot:

• node siobench.js 1.3.5 stops at ~1400 clients, RSS ~ 90Mb:
  
• node siobench.js 1.3.6 stops at only ~700 clients, RSS ~ 180Mb:
  
  (The tests were of course run on the same machine) Has anybody else noticed such a drop in performance?
  After some analysis, this seems to be related to the following commit ae5420b (you can bench it with node siobench.js 1.3.6-patch), which consists in a bump of engine.io (socketio/engine.io@ab2bd0b to socketio/engine.io@f3fd4bb, which itself consists in some minor changes (I think) and a bump of engine.io-client (socketio/engine.io-client@4f87aae to socketio/engine.io-client@d010541), which includes the following:

   "dependencies": {
-    "has-cors": "1.0.3",
-    "ws": "0.7.1",
-    "xmlhttprequest": "https://github.com/rase-/node-XMLHttpRequest/archive/a6b6f2.tar.gz",
+    "has-cors": "1.1.0",
+    "ws": "0.7.2",
+    "xmlhttprequest-ssl": "1.5.1",

Could this be related? Or are my results completely inconsistent?

Edit: Nodejs version
npm -v 2.11.3
node -v v0.12.7

[darrachequesne]

Hi! What I've found so far (not much...):

The dependencies tree

• socket.io 1.3.5 > 1.3.6
  • engine.io 1.5.1 > 1.5.2
    • ws 0.5.0 > 0.7.2
  • socket.io-client 1.3.5 > 1.3.6
    • engine.io-client 1.5.1 > 1.5.2
      • ws 0.4.31 > 0.7.2

I've created a simple Gist for ws and for socket.io, showing the memory footprint:

lib	# clients	RSS	HeapUsed	HeapTotal
ws 0.4.31	100	25239552	7471736	17029888
	1000	46194688	18566624	34705920
	2000	74207232	34065024	60758528
ws 0.5.0	100	25448448	7545976	17029888
	1000	46600192	18864336	35737856
	2000	74448896	34453288	60758528
ws 0.7.2	100	25862144	9937248	17029888
	1000	64815104	21833544	53534976
	2000	80400384	40001552	64886272

lib	# clients	RSS	HeapUsed	HeapTotal
socket.io 1.3.5	100	30367744	10808992	20113664
	1000	75706368	32486400	63830272
	2000	98512896	50888312	83388928
socket.io 1.3.6	100	52613120	12960608	30566144
	1000	200294400	35544328	68989952
	2000	339722240	63291528	90660608
socket.io 1.3.7	100	55603200	15025152	30566144
	1000	205119488	35335240	71041792
	2000	338436096	63138784	92712448

Note: Again, if anyone is able to reproduce this, or point an error in my tests, you're welcome!

There is actually an increase in memory footprint between ws 0.4.31 and 0.7.2 (~ 10%), but far lower than the one in socket.io between 1.3.5 and 1.3.6. But I am for sure missing something.

[nkzawa]

I think these are related to the permessage-deflate feature of ws which consumes a lot of memory.
Socket.io add extra informations to a message so that I thinks it uses more memory even if you send a same amount of data.
Unfortunately there is no API to disable permessage-deflate on socket.io for now.

[darrachequesne]

@nkzawa it seems to be it! I had seen this issue websockets/ws#454, but couldn't understand why setting perMessageDeflate: false didn't do anything:

// on the server side
var server = require('socket.io')();
server.listen(8181, { 'perMessageDeflate': false });
// on the client side
var client = require('socket.io-client');
var socket = client('http://localhost:' + '8181', { 'multiplex': false, transports: ['websocket'], perMessageDeflate: false });

Just to see that your two commits (socketio/engine.io@d11e17c and socketio/engine.io-client@f6bf9f8) were not merged as of version 1.3.6 and 1.3.7 of socket.io and socket.io-client. They seem to be merged on master now, so I suppose they will be released with next version.

I updated the gist accordingly. Running it after npm install socketio/socket.io socketio/socket.io-client --production (install from github#master) shows a 'normal' memory consumption (similar to 1.3.5), while enabling the perMessageDeflate feature (which is the default) actually exhibits the high consumption in memory seen above.

All right, sounds good to me!

PS: yet the hit in performance -- memory and additional processing -- makes the perMessageDeflate feature a quite expensive one (useful with lots of data and few clients, maybe).

[fourpixels]

Hah I can't belive this was it. Good job @darrachequesne !
Can I ask why this is the default behavior, if it has such a high memory consumption? And when is expected the next release to be, because I currently rely heavily on socket.io and would love not to dive in such a problems (thanks again @darrachequesne for doing it :))

[nkzawa]

This is the default behavior because ws makes it default and socket.io doesn't touch it at all for now. I'm not sure about the next release.

[darrachequesne]

I'm now closing the issue, since the memory issue has been identified thanks to @nkzawa (yet I still don't understand how a ~10% memory increase in ws library is converted to a ~250% one here, if anyone has the truth about this...)

@fourpixels I actually did nothing 😄 may I suggest you stick to the version 1.3.5 for now ?

[fourpixels]

Hm, I just revised this again. Seems like @nkzawa merged it with master on January 5th! And 10 months later it is still not merged into any working branch.. Am I missing something, or this is not planned to get done in the near future? Sorry for being such a pain in the ass, but I can't understand why is this happening?

Please re-open this one, as the issue still exists, and in fact a solution is made way back in time. Someone just needs to merge it! Otherwise it will be left just like that.. @rauchg please help us out!

[rauchg]

This fix is going into 1.5.0 due out next week!

[darrachequesne]

Great news! 👍

From what I understand (and what I've read here), the perMessageDeflate feature creates a buffer object for each client, and with many client connections it can be a big memory overhead.

A sample heap snapshot comparing with/without perMessageDeflate enabled:

Linked with:
socketio/engine.io#357
and websockets/ws#607

[Demostrike]

I'm actually seeing what I perceive to be a memory leak and I believe it to be related to this. If i run 3000 connections over some simple socket.io server code on 1.3.7 doing 200 connects/disconnects a second the resultant memory is over 1gig after about 10-15 minutes, and the memory after all the sockets disconnect is never released. (This leak seems to be in RSS memory as the heap returns to the prior size.)

The same code on 1.3.5 runs as expected, reaching a max value of ~200meg, with the whole memory footprint returning to within normal limits after they disconnect.

@rauchg When you say a fix for this is in 1.5.0 do you mean a toggle for the option or a general memory optimisation/fix?