ValueError: invalid literal for int() with base 10: b'\xff'

[MoshePerez]

I'm using python-socketio 4.3.1 and python-engineio 3.10.0
Just like in issue #39 I'm getting this error on my server:

Traceback (most recent call last):
  File ".../lib/python3.7/site-packages/engineio/server.py", line 544, in _trigger_event
    return self.handlers[event](*args)
  File ".../lib/python3.7/site-packages/socketio/server.py", line 705, in _handle_eio_message
    pkt = packet.Packet(encoded_packet=data)
  File ".../lib/python3.7/site-packages/socketio/packet.py", line 43, in __init__
    self.attachment_count = self.decode(encoded_packet)
  File ".../lib/python3.7/site-packages/socketio/packet.py", line 84, in decode
    self.packet_type = int(ep[0:1])
ValueError: invalid literal for int() with base 10: b'\xff'

Seems like there is a race condition in the _handle_eio_message function in server.py when receiving binary messages.
The function assumes that it will get the messages in the correct order (placeholder followed by the corresponding binary data) but it is not always the case.
For example, if the client emits from 2 separate threads at the same time, and the order of execution happens to be:

emit 1 sends placeholder
emit 2 sends placeholder
emit 2 sends the binary data
emit 1 sends the binary data

ValueError will be raised.

It can be reproduced easily using the python-socketio client, and adding sleep time to _send_packet function in client.py:

    def _send_packet(self, pkt):
        """Send a Socket.IO packet to the server."""
        encoded_packet = pkt.encode()
        if isinstance(encoded_packet, list):
            binary = False
            for ep in encoded_packet:
                self.eio.send(ep, binary=binary)
                ####### added for testing #####
                print("sent packet:", "<binary>" if binary else ep)
                if not binary:
                    time.sleep(5)
                ################################
                binary = True
        else:
            self.eio.send(encoded_packet, binary=False)

And emitting from 2 different threads at the same time.

[miguelgrinberg]

Have you seen this error without adding this artificial delay? I'll think about a solution, but just want to understand how to make this happen for real, because the chances of this happening are extremely small.

[MoshePerez]

@miguelgrinberg yes, I only added this delay to find out the reason for my error.
on the client side, I have an event handler that emits an image.
when the server emits this event to my client 2 or more times (to get multiple images), multiple threads will run at the same time. so the first sends the place holder, and then the binary data. but I guess since the binary data take a bit more time to upload, another thread can send a placeholder during that time.

[miguelgrinberg]

After some consideration, I've decided to just document the potential concurrency issue, and suggest applications that need to emit concurrently use a Lock object to address this problem.