print('\udcc3'): uncaught error (the GUI hangs, the error is printed in the console)

[kbulygin]

Hello! I've encountered the following error.

Steps to reproduce:

• Create a new notebook.
• Insert these lines in the empty cell:

import pathlib
print(pathlib.Path(b'Ausrei\xc3%9Fer'.decode('utf8', 'surrogateescape')))

(Simplified: print('\udcc3').)

• Press Ctrl-Enter.

Result: the UI doesn't respond (the cell blinks like when its code is being executed), Kernel > Interrupt doesn't work (creating new notebooks works).

The shell transcript for python, ipython, jupyter running the Python code:

$ python
Python 3.6.0 (default, Mar  9 2017, 15:25:26)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pathlib
>>> print(pathlib.Path(b'Ausrei\xc3%9Fer'.decode('utf8', 'surrogateescape')))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc3' in position 6: surrogates not allowed
>>>

$ ipython
Python 3.6.0 (default, Mar  9 2017, 15:25:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pathlib
   ...: print(pathlib.Path(b'Ausrei\xc3%9Fer'.decode('utf8', 'surrogateescape'))
   ...: )
   ...:
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-1-75e939ed8301> in <module>()
      1 import pathlib
----> 2 print(pathlib.Path(b'Ausrei\xc3%9Fer'.decode('utf8', 'surrogateescape')))

UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc3' in position 6: surrogates not allowed

In [2]:


$ jupyter notebook --version
5.0.0
$ jupyter notebook
[I 17:29:46.219 NotebookApp] Serving notebooks from local directory: /home/k/tmp
...
[I 17:30:00.238 NotebookApp] Adapting to protocol v5.1 for kernel 70dd7cfa-3579-45bd-a946-095efd6d0e9a
ERROR:tornado.general:Uncaught exception, closing connection.
Traceback (most recent call last):
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
    callback(*args, **kwargs)
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/ipykernel/iostream.py", line 105, in _handle_event
    event_f()
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/ipykernel/iostream.py", line 340, in _flush
    parent=self.parent_header, ident=self.topic)
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/jupyter_client/session.py", line 730, in send
    to_send = self.serialize(msg, ident)
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/jupyter_client/session.py", line 625, in serialize
    content = self.pack(content)
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/jupyter_client/session.py", line 103, in <lambda>
    ensure_ascii=False, allow_nan=False,
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/zmq/utils/jsonapi.py", line 43, in dumps
    s = s.encode('utf8')
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc3' in position 31: surrogates not allowed
ERROR:tornado.general:Uncaught exception, closing connection.
Traceback (most recent call last):
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events
    self._handle_recv()
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv
    self._run_callback(callback, msg)
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
    callback(*args, **kwargs)
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/ipykernel/iostream.py", line 105, in _handle_event
    event_f()
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/ipykernel/iostream.py", line 340, in _flush
    parent=self.parent_header, ident=self.topic)
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/jupyter_client/session.py", line 730, in send
    to_send = self.serialize(msg, ident)
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/jupyter_client/session.py", line 625, in serialize
    content = self.pack(content)
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/jupyter_client/session.py", line 103, in <lambda>
    ensure_ascii=False, allow_nan=False,
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/zmq/utils/jsonapi.py", line 43, in dumps
    s = s.encode('utf8')
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc3' in position 31: surrogates not allowed
ERROR:tornado.application:Exception in callback None
Traceback (most recent call last):
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/tornado/ioloop.py", line 888, in start
    handler_func(fd_obj, events)
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events
    self._handle_recv()
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv
    self._run_callback(callback, msg)
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
    callback(*args, **kwargs)
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/ipykernel/iostream.py", line 105, in _handle_event
    event_f()
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/ipykernel/iostream.py", line 340, in _flush
    parent=self.parent_header, ident=self.topic)
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/jupyter_client/session.py", line 730, in send
    to_send = self.serialize(msg, ident)
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/jupyter_client/session.py", line 625, in serialize
    content = self.pack(content)
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/jupyter_client/session.py", line 103, in <lambda>
    ensure_ascii=False, allow_nan=False,
  File "/home/k/.pyenv/versions/3.6.0/lib/python3.6/site-packages/zmq/utils/jsonapi.py", line 43, in dumps
    s = s.encode('utf8')
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc3' in position 31: surrogates not allowed
[W 17:30:00.985 NotebookApp] 404 GET /api/kernels/05ba95a0-2bb5-43a7-9ba6-aad9a0da3519/channels?session_id=E6D325F17E894F098643429822E0C116 (::1): Kernel does not exist: 05ba95a0-2bb5-43a7-9ba6-aad9a0da3519
[W 17:30:00.987 NotebookApp] 404 GET /api/kernels/05ba95a0-2bb5-43a7-9ba6-aad9a0da3519/channels?session_id=E6D325F17E894F098643429822E0C116 (::1) 7.03ms referer=None
[W 17:30:34.238 NotebookApp] Replacing stale connection: 05ba95a0-2bb5-43a7-9ba6-aad9a0da3519:E6D325F17E894F098643429822E0C116
...

Although it is certainly an edge case, I think it would be nice to have such unexpected errors be translated to the GUI.

[takluyver]

@minrk maybe this should be fixed in pyzmq? The RFC for JSON says that an unpaired surrogate is valid in a JSON string, though the meaning is not defined. Python uses them to roundtrip bytes that it can't correctly decode with the surrogateescape error handler.

[minrk]

Seems like there should be an error on the unpaired surrogate at some point, but probably not in the message plumbing.

Does the error go away if jupyter_client specifies ensure_ascii=True?

I'm not sure how to fix it in pyzmq, but if there is a solution I'm happy to add one. I'm not sure what to do if it's invalid UTF-8. Maybe encode('utf-8', 'replace')? That doesn't seem right, though.

[fgrosshans]

I have encountered the same problem with print('\ud800') (the 1st surrogate, an easier codepoint to remember). The cell hangs and I have to kill the kernel.
Failing with raising the UnicodeEncodeError, like the python interpreter does, seems the thing to do.

Jupyter does something similar with NameError if I ask print(a) with undefined a.

[fgrosshans]

I don’t know if it helps, but bytes('\ud800') raises the expected error. It seems that’s only print which stucks the notebook