Spawning many kernels may result in ZMQError

[JohanMabille]

Spawning many kernerls in a short lapse of time may result in ZMQError because one of the kernel tries to use a port already in use by another kernel.

This is due to the current implementation of jupyter_client: after free ports have been found, they are dumped in a connection file that will be passed to the kernel that the client will start.

The problem is that we might search for free ports after creating the connection file but before starting the kernel (when restoring a session in Jupyter Lab, or spawning multi kernels quickly in Voila for instance). Since the first kernel has not started yet, the ports are still free and jupyter_client might write a connection file for the next kernel to start with the same ports as in the first connection file. Therefore two kernels will attempt to use the same ports.

Even if we can fix this issue in Jupyter Lab and Voilà (by searching free ports for all kernels first, and then writing all the connection files at once), this does not prevent other applications (unrelated to the Jupyter project) to start and use the port written in the connection file before the kernel has started.

A solution would be to always let the kernel find free ports and communicate them to the client (kind of handshaking pattern):

• The client opens a socket A, passes the port of this socket to the kernel that it launches and waits
• the kernel starts, finds free ports to bind shell, control, stdin, heartbeat and iopub sockets. Then it connects to the socket A of the client, sends a message containing these ports, and close the connection to socket A.
• Upon reception of this message, the client connects to the kernel and closes the socket A.

I am aware that this requires significant changes in the kernel protocol and the implementation of a lot of kernels, but I do not see a better solution to this issue.

cc @vidartf and @martinRenou who have been discussing this issue in Voila

[kevin-bates]

Hi @JohanMabille - this is exactly what we do in Jupyter Enterprise Gateway - let the kernel (actually a kernel wrapper) determine its ports and return the connection info back to the server, who's listening on a port it passed to the wrapper (our term is kernel launcher) at launch time. This was primarily done to deal with kernels launched on remote hosts, but we've also had users, faced with this classic jupyter race condition, use this for local kernels as well. We use kernel launchers because we didn't want to force kernel updates for anyone using EG and remote kernels.

I haven't had a chance to look into @takluyver's kernel nanny, but I think the launchers are similar concept. The EG launchers do a bit more than just manage the ZMQ connections. They also create a Spark context (if requested) and are used to listen for interrupt and shutdown requests (since signals don't span node or user-id boundaries and graceful shutdowns don't always happen in the real world). I think the connection management and interrupt/shutdown portions could be generally useful.

[SylvainCorlay]

Thanks for the details @kevin-bates.

@martinRenou is implementing a backward compatible workaround for jupyter_server's mapping manager but I think that this is a critical-enough issue that it should trigger a conversation about changing what kernelspecs should look like (i.e. not having a connection file set by the client, but a single port for specified for the handshake socket).

Another situation in which it occurs is when jupyterlab attempts to recreate all the kernels corresponding to the open notebooks in a layout. All the kernels are opened at once and step on each other's toes with respect to ports because of that wrong logic.

[SylvainCorlay]

cc @minrk @maartenbreddels @blink1073

[maartenbreddels]

Note that I don't see the behaviour that @JohanMabille describes on OSX with voila, which could be due to OSX not returning the same 'free' port.

However, I do see the issue @SylvainCorlay describes, which is:

• Open X browser tabs (say 5) with classical notebook.
• Kill notebook server
• Restart notebook server
• Wait forever til the kernels connect.

These may be different issues, but I'm not 100% sure.

[kevin-bates]

@SylvainCorlay - yes, this is identical to what we already do in EG, so I'd like to make sure things are done in a compatible way and I'd love to see this in MappingKernelManager, although, frankly, this could be done in an even more compatible way by abstracting the subprocess (Popen) and providing hooks for implementations to provide their own launch mechanisms. One of the lower base classes would then setup the single port connection to convey to the kernel launch. This way existing kernelspecs are fine and users can migrate to using the abstraction. In EG, these are process-proxies. In the parlance of kernel providers, these are called kernel lifecycle managers, although I'm thrilled with that name either.

At any rate, should the kernel provider proposal not get "ratified", I'd like to see this abstraction layer get introduced at the Subprocess level, so folks can bring their own kind of launchers and lifecycle management. This could be done in jupyter_client and retain the existing architecture.

Fundamental to all of this, however, is async kernel management. This needs to get done in the new jupyter server!

[SylvainCorlay]

although, frankly, this could be done in an even more compatible way by abstracting the subprocess (Popen) and providing hooks for implementations to provide their own launch mechanisms

Interesting. I am not sure about different kernels having different launch mechanisms. As a xeus co-author, I like the idea of a kernel simply being an executable implementing one handshake mechanism. Ideally, people should also be able to come up with alternate implementations of jupyter_client (e.g. in C++) that could launch and talk to any kernel.

[minrk]

This is how IPython parallel works as well (although all kernel sockets connect, not just one and then binding the rest). I agree that letting kernels pick ports is best, we just need to figure out a backward-compatible mechanism to make the transition in jupyter-client. Having a listening socket in the client can make things complicated for remote kernels since we are introducing a new relationship of connectable process (the kernel manager) and process which needs to know where that is (the kernel), but the local case should not be complicated.

#490 is a great workaround for the short-term, which should help in very close to all cases.

[SylvainCorlay]

I agree that letting kernels pick ports is best, we just need to figure out a backward-compatible mechanism to make the transition in jupyter-client. Having a listening socket in the client can make things complicated for remote kernels since we are introducing a new relationship of connectable process (the kernel manager) and process which needs to know where that is (the kernel), but the local case should not be complicated.

Agreed on the need to find a smooth transition between the two approaches.

I think that the case a remote kernels is somewhat orthogonal, since kernels (as a process exposing zmq sockets) probably still have a local client. The remote handshaking mechanism can be completely different I presume.

[Carreau]

Upon reception of this message, the client connects to the kernel and closes the socket A.
I am aware that this requires significant changes in the kernel protocol and the implementation of a lot of kernels, but I do not see a better solution to this issue.

I don't see any reason to close socket A. Just keep it open for the next kernel that will start.

Hopefully I'm going to assume the client that starts the kernel knows whether this kernel will know how to do a handshake. If so we can just use a convention of placeholder value in the connection file, or an extra field.

I don't think there would be any problem with backward compatiblity.

[JohanMabille]

Hopefully I'm going to assume the client that starts the kernel knows whether this kernel will know how to do a handshake.

I guess we can add a field to the metadata section of the kernelspec (just like what we did for the debugger).

[MSeal]

I don't see any reason to close socket A. Just keep it open for the next kernel that will start.

You can't bind to a port that's already bound is the problem. So the kernel can't take the port until you release it. Once you release it another process (like a parallel notebook execution process) might bind it to hold. You can release and reconnect while it's in the TIME_WAIT state by settings the SO_REUSEADDR flag seen here: https://docs.python.org/3.3/library/socket.html to ensure you get that port in the subprocess, but there is a timing window for this that may be hard to control.