An event system for Jupyter

[afshin]

This is a draft document, please feel free to comment and help.

We have (at least) two concurrent efforts that overlap but are not full implementations of a generic event system for Jupyter in themselves: jupyter-telemetry and jupyterlab-notifications.

A synthesis of these extensions with generic endpoints (i.e., not specifically designed and named for telemetry or notifications) would yield a flexible general-purpose event bus for jupyter-server-based applications.

cc: @andrii-i @3coins

Architecture of events API

REST Endpoints

• POST /api/events - create new events
• GET /api/events/schemas - query/list registered schemas
  (maybe -- needs discussion)
• POST /api/events/schemas - register schemas
  (maybe -- needs discussion)

WebSocket endpoints (WebsocketHandler)

• /api/events/subscribe - fire hose of all events -- perhaps accept filters? (see open question below)
• /api/events/subscribe/notification -- subscribe to events of type notification

Open Question: Should the WebSocket handler support making a request for multiple filters to be applied instead of just the one proposed in the URL scheme above?

Depends on jupyter_events package

• exports EventLogger object in (formerly EventLog in jupyter_telemetry)

---

Case Study: JupyterLab Notifications

Server-side functionality

• Subscribes to all notification events that pass through the event bus
• Adds each notification as a row in a SQLite database on the server with a key for the recipient identity as well as an ID
  • notification events with multiple recipients can be de-normalized here and written as multiple rows

• REST API

  • GET /api/notifications - retrieve a list of all notifications that authenticated user can see
  • GET /api/notifications/{ID} - retrieve a specific notification
  • DELETE /api/notifications/{ID} - delete a specific notification

Client-side functionality

• Subscribe to the /api/events/notifications WebSocket
  • Throttle its incoming messages at some reasonable rate (on the order of 0.5-1 seconds)
• Treat incoming messages from the events API as a notifier only -- check the /api/notifications endpoint for the actual list of messages
• Render the badge and the notification center UI inside JupyterLab/Notebook

JupyterLab 4 extension

• A Token (e.g., INotifications or IEvents) that exposes an IDataConnector for event CRUD and an ISignal for event subscription
• A visual UI for an event notification center

Jupyter Notebook 7 extension

• The Token from the JupyterLab extension
• A version of the JupyterLab UI for notifications

[3coins]

@afshin
Should we just add this to the server or need a new server extension package? Is anyone assigned to this task?

[afshin]

@3coins, this should be in the core server.

Currently, no one is specifically assigned. I'd like to see the user interface portion of this landing in JupyterLab and I am happy to work on any part of the stack that helps get us there.

I think that the work on this already done in the telemetry space might be farther along than the server extension from the notifications extension, so grafting those handlers into jupyter-server might be the best way of bringing this into core.

What are you thinking? Let's have a conversation about this with all the people who have interest and bandwidth to work on it.

[3coins]

@afshin

What are you thinking? Let's have a conversation about this with all the people who have interest and bandwidth to work on it.

Agree, let me know if you want to have an offline discussion including anyone else who wants to work on this; personally, I would like to get some experience on the server side, but happy to work on any part of the stack. Is there an expected time frame to get these changes done?

[afshin]

I think Zach is rounding up interested folks (including you) for a conversation.

We are targeting jupyter-server v2 and jupyterlab v4 (so late June, early July).

[rahul26goyal]

@afshin / @Zsailer :
please include me on any meeting that might happen related to this. I am interested to learn more on this area and contribute anyway I can.

[3coins]

As discussed in the server meeting on 5/5/2022, here is an initial list of tasks for the event notification system. This list is by no means final, feel free to add comments or feedback.

1. Event Bus - #820

   • A central event bus to relay events
   • /api/events/subscribe - Websocket for subscribing to events
   • A default handler for consuming events

2. Rest API Endpoints

   • POST /api/events - Rest api to create new events
   • GET /api/events/schemas - Rest api to query/list registered schemas (Optional)

3. Event buffer

   • A queue/buffer to store undelivered event messages

4. JupyterLab 4 Event Client (jupyterlab-events)

   • Reuse jupyterlab-telemetry repo, either rename or copy to jupyterlab-events
   • Remove server endpoints, any redundant server code
   • Update client handlers to use the rest api endpoints
   • Add websocket handler to enable subscribtion to events

5. Add Default events in server

   • Add default events e.g., content handler, kernel events in jupyter server

6. JupyterLab 4 Updates

   • Add jupyterlab-events as dependency inside JupyterLab
   • Subscribe to default events

7. Event Notification UI (JupyterLab)

   • UI updates for event notification

8. Jupyter Notebook 7 Updates

   • Add jupyterlab-events as dependency
   • Subscribe to default events
   • Can we reuse event notification UI from JupyterLab?

[afshin]

Here is a document we can collaboratively edit so that the front-matter of this issue can have a canonical version that we edit once it is ready: https://hackmd.io/q4Rkq2BaS1SIXvyzt8j1yA

[davidbrochart]

Since the event system is a new service that we are just starting to develop, how about making it as much as possible backend-agnostic? By that I mean that most of the logic should be usable in both jupyter-server and jupyverse.
But it is currently very tied to jupyter-server, Tornado and traitlets, which we don't want to depend on in jupyverse.