Possibility to support ipykernel

[saulshanabrook]

We have a client who is interested in this work and got this feedback:

This looks very, very promising. I noticed it requires xeus-python. Do we know if they plan to support the ipykernel in the long run? While I have no objection to the xeus kernel in theory, in practice it seems to both be not be popular (73 stars after over a year of development) nor have all features of the ipykernel. It would seem one or the other would have to give for this to be feasible for us.

So what would we have to add to ipykernel to make it compatible? And do you see any large blockers in implementing that support?

[jtpio]

It would be great to have support for it yes.

To make it compatible ipykernel will have to support the Debug Adapter Protocol, for example by using ptvsd (like xeus-python). As well as a couple of other messages that are not part of the DAP, such as debugInfo (to retrieve the state of the current debug session) and dumpCell (to debug cells).

Probably the main obstacle would be to make sure ipykernel is able to process messages from the control channel in a concurrent manner while shell is blocked (for example when the execution is stopped after hitting a breakpoint).
The control channel is used to send and receive the debug messages: https://jupyter-client.readthedocs.io/en/latest/messaging.html#debug-request

There is an overview of the debug protocol in #64 (comment). The plan is to add more information to it and make an official documention so it's easier for other kernels to start adding support for debugging.

For the JupyterLab extension, this shouldn't require any change.

[saulshanabrook]

Probably the main obstacle would be to make sure ipykernel is able to process messages from the control channel in a concurrent manner while shell is blocked (for example when the execution is stopped after hitting a breakpoint).

Does anyone know if there is an existing issue for this open in ipykernel or have more insight into how much work this would take? cc @Zsailer @Carreau

[SylvainCorlay]

There are some intrinsic limitations with ipykernel which may require either the architecture to be substantially changed, or the experience of the debugger to be degraded.

Also, I don't think that the popularity is a factor at all here. xeus-python or the debugger have not been advertised widely - and I don't think that feature parity is out of reach given the current coverage of the kernel feature.

[saulshanabrook]

So if we assume that the debugger will grow to be a core feature in JupyterLab that we want to enable for folks, then it seems to suggest that we either:

1. Re-architect ipykernel to support this message processing issue so we can add debugger support
2. Promote xeus-python instead of ipykernel as the default Python kernel (possibly leading to it shipped by default instead the ipykernel) and make sure it has feature parity with ipykernel

Although at the moment this is just a debugger conversation, it seems like the implications here are far reaching in terms of what we choose to support going forward with JupyterLab. Ideally, it would be nice to know which direction to move before we invest substantial resources.

[SylvainCorlay]

Promoting xeus-python instead of ipykernel would be controversial, but there is an ongoing JEP on putting xeus and related project under the Jupyter governance in a Jupyter-affiliated GitHub organization.

This would be the case for xeus, xeus-python, xeus-cling, and some related utility projects.

So Xeus will become a reference implementation of the Jupyter protocol, and include the first implementation of the DAP.

[SylvainCorlay]

I would argue that xeus is a safe bet for broader adoption. QuantStack is funded for a little while to continue pushing on the project and downstream projects (the debugger, xeus-python, etc).

Since it is a reboot of the kernel implementation, it has less bagage and already has a broad coverage of the features with a much lighter-weight codebase.

[SylvainCorlay]

Also, regarding the inclusion into JupyterLab, I don't think that it requires that ipykernel supports the protocol, or that xeus-python is "the official kernel". JupyterLab is a kernel agnostic front-end. It can advertize the debugger features for kernels that support the debugger protocol.

[timkpaine]

As an end user (and non-conda user), xeus-python has always been difficult to build and integrate. The matrix of dependency compatibilities is confusing, and compilation on centos 6 has not been straightforward. If there was a manylinux1 wheel published on pypi, i'd probably have a different opinion (and obviously we can make some of these things more user friendly).

[SylvainCorlay]

As an end user (and non-conda user), xeus-python has always been difficult to build and integrate. The matrix of dependency compatibilities is confusing, and compilation on centos 6 has not been straightforward. If there was a manylinux1 wheel published on pypi, i'd probably have a different opinion (and obviously we can make some of these things more user friendly).

Issue jupyter-xeus/xeus-python#181 tracks progress on building a PyPI wheel. Beyond the need for a wheel, do you have specific feedback on improving the build process for xeus-python?

[Carreau]

I'd love to have the protocol implemented in ipykernel; I understand that it might be difficult to have the same experience as Xeus-Python; One thing I want to avoid as much as possible is having a confusing story.

If users needs to switch between ipykernel and Xeus-python depending on wether they want to use a given magic or the debugger the experience will be sub-par. If we have to choose, a clear distinction of features would be better than two almost similar projects.

On the note of popularity; I believe this is likely due to which kernel is shipped by default and the popularity of Xeus-Python would likely go up quite a bit if we start to ship it by default with conda for example.

[timkpaine]

@SylvainCorlay Not really, the biggest headache with these types of packages are always that either the wheel has a glibc dependency that wont work on centos 6, or that the source build either tries to bring in the whole world without letting me tell it that i already have slightly different but compatible versions of the dependencies.

The fact that i can build and separately control all the dependencies is great, though the docs only suggest installing them with conda. We could just quickly make a wiki to document some of the bugs that pop out if you go to install every dependency from source, most of which I'm sure you already stumbled upon when you packaged all this stuff up for conda.