Unicode Parsing Issue

[dominiek]

I've discovered a serious issue in the latest Socket.IO (0.7.7). When certain unicode (6.0) characters are sent through the socket, Socket.IO's internal parser will fail and the event will never arrive properly to the server. I've tested it on the websocket transport in Chrome, but it seems to also fail on other transports.

Here is a piece of example code that recreates it:

  var socket = io.connect('http://localhost/endpoint');
  socket.emit('any-event', {messages: [{"text":"@nov Nice presentation \ud83d\udc4d"}]}, function(err, ) {
    // this will never return and the above data will never ariive
  });

The character in question here is the unicode "thumbs up sign" http://www.fileformat.info/info/unicode/char/1f44d/index.htm - However, I've seen this fail on other pieces of unicode too! This is a serious issue guys.

More details on the situation:

On the client-side, the last 3 characters of the JSON string are as follows when being sent on the websocket (https://github.com/LearnBoost/socket.io-client/blob/master/dist/socket.io.js#L2322 ):

  32 // the space after the word "presentation"
  55357
  56397

However, on the server-side (https://github.com/LearnBoost/socket.io/blob/master/lib/transports/websocket.js#L132) this arrives as follows:

  32 // the space after the word "presentation"
  65533

Please note that 65533 is the same as '\ufffd', the special character that's used for parsing.

[3rd-Eden]

I would love to have a failing test case for this. As our unicode test: https://github.com/LearnBoost/socket.io-client/blob/master/test/socket.test.js#L338-358 passes

[dominiek]

Whoop there it is! socketio/socket.io-client#275

[dominiek]

(make sure to run the test-suite in websocket transport mode)

[dominiek]

Please note that this is also happening from Server to Client. This bug can be exploited as a denial of service attack, so I think we should mark this as critical.

I've added a test here to illustrate the issue: socketio/socket.io-client#275

My suggestion for a fix is choosing a different sequence of bytes for the framing of socketIO packets.

[dominiek]

Any updates on this? I have a pending pull-request for a unit test that illustrates this

[frank06]

subscribing

[3rd-Eden]

Just got back from vacation today and will look at it once Node Knockout is finished.

[Pita]

I have a short example to reproduce that, if you execute this command on the client side

socket.json.send({test: "\u2028"});

the message gets dropped at this line https://github.com/LearnBoost/socket.io/blob/master/lib/parser.js#L145 on the server side

[Pita]

We should output a warning when a message gets dropped on the client side

[rauchg]

Fixed! Please upgrade to 0.8.3

[dominiek]

Awesome! Can't wait

[corwin-of-amber]

Hello! I've noticed that the ({test: "\u2028"}) case works fine indeed, but the "thumbs up" case "\ud83d\udc4d" causes the socket to actually disconnect. Using Socket.IO 0.8.7 right now.

[dominiek]

We have also noticed that it still fails on some unicode combinations.

We have a hack in our code to deal with this (this is only possible because we are not dealing with mission critical messages yet). All our messages sent down to the client and up to the server are going to a hacky filter like this:

  var _filterUnicode = function(messages) {
    if (!messages || !messages.length) return messages;
    var prevCount = messages.length
      , expr = /[^\u0000-\ud83d]/;
    messages = messages.filter(function(m) {
      return !(m.title && expr.test(m.title));
    });
    var stripped = prevCount - messages.length;
    if (stripped) console.debug("Stripped out "+stripped+" messages due to SocketIO unicode bug");
    return messages;
  }