Towards a new generic and composable server

[SylvainCorlay]

The current jupyter_server project started as a split of the backend parts of the notebook repository, and the classic notebook front-end is now installable as a separate package providing a server extension https://github.com/jupyterlab/nbclassic, and JupyterLab has adopted the new package.

While jupyter_server has changed quite a bit from the original notebook backend, it still includes a lot of the history of the original project.

Problems with the current server

Tornado

IIRC, Tornado was one of the earliest Python web servers to support WebSockets, and provided a modern async programming model long before asyncio existed. It was adopted broadly in the Jupyter stack, so much that e.g. ipykernel and jupyter_client depend on Tornado…

However, in my opinion, Tornado has become a liability:

• The async constructs are a compatibility layer on top of asyncio. However, there are some tricky corner cases that make it not great to work with.
• Several of the recent versions of Tornado broke Jupyter in subtle ways, and we've had to patch various Jupyter subprojects to accommodate new versions.
• It is not so actively developed compared to alternatives.

Dropping Tornado and building a new server on top of another stack would be a complete reboot of the project - and would not allow any existing server extension to be used with it.

The current HTTP and WebSocket APIs

HTTP endpoints

The current HTTP endpoints could be improved in several ways. For example, we could work on

• providing pagination for endpoints returning large amounts of data (such as the content API in directories listing too many files).
• providing hooks in the content API so that we can handle e.g. lazy loading of large files instead of always sending the full content in the front-end.
• breaking the coupling and the tight assumptions that are made between certain HTTP endpoints.

The API could also handle certain long-running requests differently, for example by returning immediately with a token that can be used to poll another endpoint for the result.

The kernel protocol over WebSocket is inefficient

Another issue with the way the Jupyter server works is due to the way we communicate with kernels over WebSockets. The main issue in my opinion, while ZMQ messages are serialised in a well-specified sequence of blobs of bytes,

[
  b'u-u-i-d',         # zmq identity(ies)
  b'<IDS|MSG>',       # delimiter
  b'baddad42',        # HMAC signature
  b'{header}',        # serialized header dict
  b'{parent_header}', # serialized parent header dict
  b'{metadata}',      # serialized metadata dict
  b'{content}',       # serialized content dict
  b'\xf0\x9f\x90\xb1' # extra raw data buffer(s)
  ...
]

the WebSocket protocol communicates this content as a JSON object with keys for header, content, metadata etc. A consequence of that design is that all messages have to be parsed so that we can recompose the ZMQ messages.

If the WebSocket messages contained the same binary blobs as the ZMQ messages, we could directly route them to the right kernel (simply adding ZMQ identities and delimiter)... Such an approach would result in a considerably faster handling of kernel messages.

A multi-user "single instance" server

In cloud deployments, (especially with the new RTC features of JupyterLab), we will probably want have some preferences (currently configured via traitlets configurables) to become user-specific and be saved in a data base.

For examples, themes, workspaces in JupyterLab should probably be set on a per-user basis.

A proposal for a new server

Drop Tornado and reboot the Jupyter server project with a FastAPI-based solution.

Using FastAPI will come with many benefits such as modern tooling (type annotation, automatic generation of OpenAPI specs, a rich collection of tools for telemetry, authentication.

Adopt the “everything is a plugin” approach of JupyterLab to the architecture

Starting from an "empty" base server and a collection of plugins for HTTP endpoints may have important benefits compared to the current approach where the base server provides a number of endpoints already.

• we could more easily provide alternative implementations of standard endpoints.
• we could make "remixes" of the base plugins, cherry picking some endpoints from the core server and others from third-party plugins. An example use case could be a "kernels-only" server that would only provide the end-points for communicating kernels. This could be an interesting way to deal with remote servers (possibly by enabling other plugins for e.g. LSP etc).

Prototype implementation

In the past few weeks, @adriendelsalle and @davidbrochart have been working on prototyping such an approach

• https://github.com/adriendelsalle/fps provides the base server and a simple plugin system
• https://github.com/davidbrochart/jupyverse provides a collection of fps plugins for the main Jupyter functionalities. It can serve JupyterLab, retrolab, and other Jupyter front-ends.

There is still a lot to figure out naturally:

• We've had several conversation on whether we would like e.g. the base server to always require a database, and plugins to require some tables etc in this database. For plugins having special requirements, they could always require another one...

  Presumably, SQLite could be used for the case of a single machine deployment of Jupyter where the users simply types jupyter lab to launch it, but a database running on a separate machine would presumably be specified in the case of cloud deployments. (Using the same database for several plugins would help simplify the configuration).

• When discussing the project, we have also been thinking about the articulation between the single-instance server and the hub with respect to authentication and authorization. One idea that came out was to "elect" OIDC as the default authentication method, and use the hub as an OIDC identity provider in the case of hub-based deployments.

• And naturally the whole question of the transition should we move forward.

Jupyter_server

We would like to discuss those ideas with the broader jupyter_server community, and improve the proposal and the ongoing work based on the group's feedback.

Obviously, we should continue improving jupyter_server, but there are many projects that could already benefit from the proposed approach.

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[SylvainCorlay]

cc @adriendelsalle @davidbrochart @JohanMabille @jtpio @fcollonval.

[echarles]

Thx for opening the discussion.

...the articulation between the single-instance server and the hub with respect to authentication and authorization

Does https://github.com/adriendelsalle/fps support multiuser or is it like today single-user jupyter server based on tornado?

. One idea that came out was to "elect" OIDC as the default authentication method,

Sound good to me.

... in the case of hub-based deployments.

There are other deployments that don't use jupyterhub. They should also be considered, or at least make sure it is still possible to run with something else than jupyterhub.

And naturally the whole question of the transition should we move forward.

I am wondering how this would impact e.g. enteprise gateway cc/ @kevin-bates

Obviously, we should continue improving jupyter_server, but there are many projects that could already benefit from the proposed approach.

Having a dual offering is the less comfortable solution.

[SylvainCorlay]

Does https://github.com/adriendelsalle/fps support multiuser or is it like today single-user jupyter server based on tornado?

The first goal was to get a working (lean and mean) set of fps plugins for the Jupyter endpoints before we tackle more important refactors.

Ultimately, we should decide what is considered a user setting (to be saved on the database) and what is server configuration.

One idea that came out was to "elect" OIDC as the default authentication method,

Sound good to me.

🎉 great!

I am wondering how this would impact e.g. enteprise gateway cc/ @kevin-bates

Yeah it is definitely a complete reboot of the server project. However we should work together and use this occasion to be as close as possible to "the right thing" for everyone.

[SylvainCorlay]

There are other deployments that don't use jupyterhub. They should also be considered, or at least make sure it is still possible to run with something else than jupyterhub.

Yeah, absolutely. What I had in mind when I wrote this is that all it needs basically is an OIDC provider, and that the Hub is just one instance... We are actually preparing a project for a specific deployment that probably won't be based on JupyterHub so that is definitely something we have in mind...

[echarles]

The first goal was to get a working (lean and mean) set of fps plugins for the Jupyter endpoints before we tackle more important refactors.

My question was more: does fps foundation support today out-of-the box multi-user, and if not, what would it take to get that in?

[davidbrochart]

FPS is agnostic to the server API.
The Jupyter server functionality is implemented in jupyverse. It has initial support for multi-user.

[echarles]

Thx for clarification. What about the support of the existing server extensions https://jupyter-server.readthedocs.io/en/stable/developers/extensions.html (and the shim for the previous notebook server extensions implemented by nbclassic https://github.com/jupyterlab/nbclassic) ?

[davidbrochart]

Server extensions would need to be adapted to the new extension mechanism. They will typically be implemented as FPS plugins.

[kevin-bates]

I am wondering how this would impact e.g. enteprise gateway cc/ @kevin-bates

Yeah it is definitely a complete reboot of the server project. However we should work together and use this occasion to be as close as possible to "the right thing" for everyone.

This looks interesting - thank you for raising the discussion! Looking at the kernel_server plugin (in jupyverse) (which I'm assuming is a quick POC), I believe we will need to support variations of kernel launching in order to support resource-managed kernels and that needs to be accomplished in a pluggable manner. Since the kernel provisioning design is predicated on the Popen abstraction, I suspect provisioners can be usable in this new server model and would like to see that be a design goal.

I really like the FPS approach and is something we've touched on in the past in Service Composition.

Would it be possible to have some form of meeting where basic requirements could be discussed and a roadmap possibly outlined? This seems to be happening fast and I think it would be good for folks to get on the same page before we're too far down the road.

[SylvainCorlay]

Would it be possible to have some form of meeting where basic requirements could be discussed and a roadmap possibly outlined?

Absolutely. We would like to discuss it at server meetings and all relevant venues. We can also schedule dedicated discussions.

kernel_server plugin (in jupyverse) (which I'm assuming is a quick POC)

It is definitely a POC since we need to enable the new WebSocket protocol, etc.

[adriendelsalle]

My question was more: does fps foundation support today out-of-the box multi-user, and if not, what would it take to get that in?

FPS is agnostic to the server API.

Sorry to answer later, but yes as @davidbrochart mentioned FPS is just a way to compose a server from different plugins relying on pluggy (from pytest project) to declare hooks specs and implementations.
It also handles configuration based on pydantic, CLI using typer, and relies on uvicorn to run the server.

Basic collision detection has been implemented to prevent multiple declaration of a route, but still a lot has to be done to handle complex scenario: multiple back-ends for a single API, compatibility rules, etc.

I really like the FPS approach and is something we've touched on in the past in Service Composition.

This discussion is awesome, I'll take a deep look at it thanks @kevin-bates !

[Zsailer]

Hey @SylvainCorlay, thanks for starting this discussion! Let's chat about it at this week's server meeting. I'll add it to the agenda.

[Carreau]

Just thinking of Realtime, widgets, etc. Would it make sens to have a server not in Python on which the notebooks models are, and not actually propagate the kernel messages all the way to the browser ? I guess in many cases even the widget code could thus also run on server side. It would also make partial rendering on the browser side easier I believe.

[SylvainCorlay]

Hi Matthias, It turns out that yjs now has a native (rust) port with Python bindings so the yjs fps plugin will eventually hold an instance of the collaborative model in the backend.

…

On Wed, Sep 8, 2021, 22:37 Matthias Bussonnier ***@***.***> wrote: Just thinking of Realtime, widgets, etc. Would it make sens to have a server not in Python on which the notebooks models are, and not actually propagate the kernel messages all the way to the browser ? I guess in many cases even the widget code could thus also run on server side. It would also make partial rendering on the browser side easier I believe. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASJOFUXSEYF4XJO3FEXPF3UA7CR3ANCNFSM5DPHGG7Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.