feat: Run exec_request inside a task

[fleming79]

This PR modifies the kernel to run exec_requests inside a task using Memory object streams to queue pending tasks. This means that other shell messages can be processed while the execution request is being processed.

ref: jupyterlab/jupyterlab#17363 (comment)

New features

• Adds the start_soon method to the kernel providing a thread safe equivalent to anyio.start_soon.

Breaking changes

• set_parent & get_parent methods are removed. Instead the parent message is only set for the current execute request providing better certainty for inputs and outputs

Known issues

• Tests pass locally but occasionally fail (need help)

Example

Screenshot

Before

ipykernel.demo.before.mp4

After

ipykernel.demo.after.mp4

Notebook code

Requires:

• ipywidgets
• jupyterlab

import anyio
import time
import asyncio
from anyio import sleep
from ipywidgets import HTML, IntSlider, __version__

slider = IntSlider()
print('ipywidgets', __version__)

# async def run_till_100():
async def run_till_100():
    display(slider)
    reported_value = HTML()
    display(reported_value)
    cval = -1
    while True:
        time.sleep(0.1) # not a good idea as it blocks the main thread
        await sleep(0.1)
        if cval != slider.value:
            cval = slider.value
            reported_value.value = f"Reported value: {cval}"
            print(cval)
        if cval == 100:
            print('Done')
            await anyio.sleep(0.7)
            slider.value = 0
            await anyio.sleep(0.01)
            return 'Done'

await run_till_100()

await asyncio.create_task(run_till_100())

shell = get_ipython()
shell.kernel.start_soon(run_till_100)

[davidbrochart]

Thanks @fleming79.
That's going in the direction of akernel, so I can only approve :)
One important thing though is kernel interrupt and how execute requests are cancelled (I don't see anything around that in the code).

[fleming79]

Thanks @fleming79. That's going in the direction of akernel, so I can only approve :) One important thing though is kernel interrupt and how execute requests are cancelled (I don't see anything around that in the code).

@davidbrochart - thank you for the review and positive feedback!

I've updated the code so it will now cancel all requests where the submitted (monotonic) time is prior to the aborted time.

[fleming79]

Here is a demo video showing a forked version of ipylab using await extensively.

The modified version of ipylab defines async functions to communicate with the frontend via custom widget messages (Widget.on_msg(...)).

ipylab.async.everywhere.demo.mp4