Notebooks don't connect to kernel when run via gateway

[moczkowski]

Since last week I've had all kinds of trouble using Datalab via a gateway VM. So today I tried from scratch. Using the latest images with a brand-new gateway VM following the quickstart instructions exactly and running on a Mac with the latest Docker installed, any notebook I open fails to connect to a kernel. Running Datalab locally works perfectly fine.

In the debug output there is an error each time:
{"name":"app","hostname":"1a994dcf2881","pid":202,"type":"jupyter","level":30,"msg":"[9000]: 200 GET /api/contents/datalab/docs/intro/Introduction%20to%20Notebooks.ipynb/checkpoints?_=1481647861344 (127.0.0.1) 7.96ms\n","time":"2016-12-13T16:51:01.985Z","v":0} {"name":"app","hostname":"1a994dcf2881","pid":202,"type":"jupyter","level":30,"msg":"[9000]: Unhandled error in API request\n Traceback (most recent call last):\n File \"/usr/local/lib/python2.7/dist-packages/notebook/base/handlers.py\", line 481, in wrapper\n result = yield gen.maybe_future(method(self, *args, **kwargs))\n File \"/usr/local/lib/python2.7/dist-packages/tornado/gen.py\", line 1015, in run\n value = future.result()\n File \"/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py\", line 237, in result\n raise_exc_info(self._exc_info)\n File \"/usr/local/lib/python2.7/dist-packages/tornado/gen.py\", line 1021, in run\n yielded = self.gen.throw(*exc_info)\n File \"/usr/local/lib/python2.7/dist-packages/notebook/services/sessions/handlers.py\", line 75, in post\n type=mtype))\n File \"/usr/local/lib/python2.7/dist-packages/tornado/gen.py\", line 1015, in run\n value = future.result()\n File \"/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py\", line 237, in result\n raise_exc_info(self._exc_info)\n File \"/usr/local/lib/python2.7/dist-packages/tornado/gen.py\", line 270, in wrapper\n result = func(*args, **kwargs)\n TypeError: create_session() got an unexpected keyword argument 'name'\n","time":"2016-12-13T16:51:01.987Z","v":0} {"name":"app","hostname":"1a994dcf2881","pid":202,"type":"request","level":30,"url":"/api/sessions","method":"POST","status":500,"msg":"","time":"2016-12-13T16:51:01.988Z","v":0} {"name":"app","hostname":"1a994dcf2881","pid":202,"type":"jupyter","level":30,"msg":"[9000]: {\n \"Origin\": \"http://localhost:8081\", \n \"Content-Length\": \"127\", \n \"Accept-Language\": \"en-US,en;q=0.8\", \n \"Accept-Encoding\": \"gzip, deflate, br\", \n \"Connection\": \"close\", \n \"Accept\": \"application/json, text/javascript, */*; q=0.01\", \n \"User-Agent\": \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36\", \n \"Host\": \"localhost:8081\", \n \"X-Requested-With\": \"XMLHttpRequest\", \n \"Referer\": \"http://localhost:8081/notebooks/datalab/docs/intro/Introduction%20to%20Notebooks.ipynb\", \n \"Cookie\": \"_ga=GA1.1.169860604.1481054659\", \n \"Content-Type\": \"application/json\"\n }\n500 POST /api/sessions (127.0.0.1) 2.21ms referer=http://localhost:8081/notebooks/datalab/docs/intro/Introduction%20to%20Notebooks.ipynb\n","time":"2016-12-13T16:51:01.989Z","v":0}

[ojarjur]

Thanks for reporting this.

I was able to reproduce it, and this seems to be an issue in a release we put out on December 10, which we've now rolled back.

We'll keep this bug open until we've found the root cause and fixed it.

[qimingj]

If local notebook is pinned to 4.2.3 it works. 4.3.0 doesn't work. There might be a bug in 4.3.0 when connecting to kernel gateway, or we don't configure jupyter correctly.

[ojarjur]

It looks like this PR added 'type' and 'name' fields to sessions, but the nb2kg extension's SessionManager has not been updated to support that yet.

This means that kernel gateways won't work with 4.3.0 until the nb2kg extension is updated.

[qimingj]

Thanks for digging this. Yes. We'll have to pin our notebook to 4.2.3.

[robertlacok]

I am trying to set up Datalab with kernels on GCP for the first time and running into problems. I'm following the quickstart, and when I navigate to the localhost:8081 and click Allow on the authentication window, browser shows ERR_EMPTY_RESPONSE and console outputs Failed to log in to gcloud.

Could it be related to this issue or should I create a new one?

[yebrahim]

The latest release is working fine for me. Is this using the latest release or a custom image? Please open a new issue and post as much info as you can get.

[yebrahim]

I opened #1092 to track this.

[tylere]

@ojarjur it looks like the nb2kg Session Manager has been updated to support the 'name' and 'type' fields. Will this allow the notebook version to move past 4.3.0?

[ojarjur]

@tylere This issue is what caused us to pick version 4.2.3, but it's no longer preventing us from upgrading.

That is because all of the gateway support has been removed.