sessionmanager KeyError when restarting kernel

[parente]

One of our internal users received the following traceback when restarting a metakernel based kernel via the Kernel -> Restart menu item. The traceback appeared in a popup dialog over the notebook page.

Notebook 5.1.0 final running in Python 3.6.

Traceback (most recent call last):
  File "/opt/jupyterhub/envs/notebook-1506165706/lib/python3.6/site-packages/notebook/base/handlers.py", line 540, in wrapper
    result = yield gen.maybe_future(method(self, *args, **kwargs))
  File "/opt/jupyterhub/envs/notebook-1506165706/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/opt/jupyterhub/envs/notebook-1506165706/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/jupyterhub/envs/notebook-1506165706/lib/python3.6/site-packages/tornado/gen.py", line 1069, in run
    yielded = self.gen.send(value)
  File "/opt/jupyterhub/envs/notebook-1506165706/lib/python3.6/site-packages/notebook/services/sessions/handlers.py", line 69, in post
    model = yield gen.maybe_future(sm.get_session(path=path))
  File "/opt/jupyterhub/envs/notebook-1506165706/lib/python3.6/site-packages/notebook/services/sessions/sessionmanager.py", line 170, in get_session
    return self.row_to_model(row)
  File "/opt/jupyterhub/envs/notebook-1506165706/lib/python3.6/site-packages/notebook/services/sessions/sessionmanager.py", line 209, in row_to_model
    raise KeyError
KeyError

[blink1073]

FWIW I tried with Octave Kernel (a metakernel), and it worked fine for me.

[minrk]

What prompted the restart? Is there a chance that the kernel had died? Did they perhaps request restart multiple times in quick succession?

Hunch without looking at the code: kernel could have died and the session been deleted, then restart could raise KeyError (though I would expect a 404).

[danielkza]

Just got the same error when restarting a jupyter-scala kernel. Similar setup to @parente: Notebook 5.1.0, Python 3.6 from Anaconda 5.0.0.1.

[parente]

What prompted the restart? Is there a chance that the kernel had died? Did they perhaps request restart multiple times in quick succession?

From the anecdotal evidence I have: kernel was still alive and the user may have done a restart twice in quick succession.

[glortho]

We get this error periodically when first navigating to a new notebook. It will show "Kernel Error" with the bomb icon. Logs show this:

[E 14:13:17.077 NotebookApp] Uncaught exception POST /notebook/api/sessions (10.255.0.4)
    HTTPServerRequest(protocol='http', host='<our host here>', method='POST', uri='/notebook/api/sessions', version='HTTP/1.1', remote_ip='<our remote ip>')
    Traceback (most recent call last):
      File "/anaconda/envs/py3/lib/python3.6/site-packages/tornado/web.py", line 1543, in _execute
        result = yield result
      File "/anaconda/envs/py3/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
        value = future.result()
      File "/anaconda/envs/py3/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
        yielded = self.gen.send(value)
      File "/anaconda/envs/py3/lib/python3.6/site-packages/notebook/services/sessions/handlers.py", line 67, in post
        model = yield gen.maybe_future(sm.get_session(path=path))
      File "/anaconda/envs/py3/lib/python3.6/site-packages/notebook/services/sessions/sessionmanager.py", line 170, in get_session
        return self.row_to_model(row)
      File "/anaconda/envs/py3/lib/python3.6/site-packages/notebook/services/sessions/sessionmanager.py", line 209, in row_to_model
        raise KeyError
    KeyError
[W 14:13:17.078 NotebookApp] Unhandled error

Compared to a healthy kernel startup log, it appears these lines are missing from this unhealthy kernel log:

[D 14:12:53.045 NotebookApp] Starting kernel: ['/anaconda/envs/py3/share/jupyter/kernels/python3/kernel-startup-py3.sh', '-f', '/home/gremlin/.local/share/jupyter/runtime/kernel-9c168b83-dd97-4e44-824c-1fe0eb628874.json']
[D 14:12:53.049 NotebookApp] Connecting to: tcp://127.0.0.1:53463
[D 14:12:53.049 NotebookApp] Connecting to: tcp://127.0.0.1:46064
[I 14:12:53.051 NotebookApp] Kernel started: 9c168b83-dd97-4e44-824c-1fe0eb628874
[D 14:12:53.051 NotebookApp] Kernel args: {'kernel_name': 'python3', 'cwd': '/home/gremlin'}

When refreshing the page, everything else is back to normal, as if it were a race condition between the sessions POST and the kernel startup lifecycle itself.

Has anyone seen the same?

Jupyter 5.6.0 on nvidia/cuda:9.0-cudnn7-runtime-ubuntu16.04

[jni]

Was this fixed in #4219? It seems like #4209 was a duplicate of this.

Anyway, just encountered this when launching a Binder notebook at https://github.com/scikit-image/skimage-tutorials. The notebook version there appears to be 5.7.8. Has #4219 made it to that release?

[kevin-bates]

Although the fix in #4219 was to accommodate culled kernel behavior, a nicer error message should be produced regardless - which you are not receiving. Since culling was, I believe, implemented after this issue was created, I do not believe its necessarily a duplicate - especially since it doesn't appear your kernel was culled.

FWIW, I have confirmed that #4219 is not in 5.7.8.