Jupyter Server Notes 2020

[Zsailer]

Hello everyone,

Welcome to the Jupyter Server Team!

You can find us in our Gitter chat room or visit us at our weekly meetings!

We meet on Thursdays at 8:00am, Pacific Standard Time on Jupyter's Zoom Channel. You can add yourself to the weekly agenda here. Everyone is welcome!

Let's avoid using this thread for discussion. If you'd like to discuss something in the minutes, open a separate issue and reference this thread.

Meeting Minutes

• December 17th, 2020
• December 10th, 2020
• December 3rd, 2020
• November 26th, 2020 - no meeting (US Holiday - Thanksgiving)
• November 19th, 2020
• November 12th, 2020
• November 5th, 2020
• October 29th, 2020
• October 22nd, 2020
• October 15th, 2020 - no meeting for JupyterCon 2020
• October 8th, 2020 - no meeting for JupyterCon 2020
• October 1st, 2020 - no minutes captured (sorry about that)
• September 24th, 2020
• September 17th, 2020
• September 10th, 2020
• September 3rd, 2020
• August 27th, 2020
• August 20th, 2020
• August 13th, 2020
• August 6th, 2020
• July 30th, 2020
• July 23rd, 2020
• July 16th, 2020
• July 9th, 2020
• July 2nd, 2020
• June 25th, 2020
• June 18th, 2020
• June 11th, 2020
• June 4th, 2020
• May 28th, 2020
• May 21st, 2020
• May 14th, 2020
• May 7th, 2020
• April 30th, 2020
• April 23rd, 2020
• April 16th, 2020
• April 9th, 2020
• April 2nd, 2020
• March 26th, 2020
• March 19th, 2020
• March 12th, 2020
• March 5th, 2020
• February 27th, 2020
• February 20th, 2020—meeting canceled.
• February 13th, 2020
• February 6th, 2020
• January 30th, 2020
• January 16th, 2020
• January 9th, 2020
• January 2nd, 2020
• December 12th, 2019—no meeting.
• December 5th, 2019
• November 21st, 2019
• November 7th, 2019
• October 31st, 2019
• October 24th, 2019
• October 17th, 2019
• October 10th, 2019

[Zsailer]

October 10th, 2019

Attendees

Add your name to the table below:

Name	affiliation	username
Zach Sailer	Jupyter Cal Poly	@Zsailer
Eric Charles	Datalayer	@echarles
Steven Silvester	J.P. Morgan Chase	@blink1073
Vidar T Fauske	J.P. Morgan Chase	@vidartf
Kyle R Kelley	Netflix	@rgbkrk
M Pacer	Netflix	@mpacer
Carol Willing	Project Jupyter	@willingc
Kevin Bates	IBM	@kevin-bates
Luciano Resende	IBM	@lresende
Safia Abdalla	Microsoft	@captainsafia

There were other attendees at the meeting, but they were not added to the list.

Agenda + Minutes

• Background [name=@Zsailer]

  • Proposal: separate the Python server from the classic Notebook frontend.

    • Jupyter Enhancement Proposal
    • Jupyter Server Documentation

  • Motivation: make all Jupyter frontends equal citizens in the eyes of the server.

    • All frontends are a server extension.
    • n frontends to 1 server.

  • Positive Side-effects:

    • Server can evolve for next generation frontends without being constrained by classic notebook (e.g. see Kernel Provider proposal).
    • Jupyter Server becomes a replaceable component. We can explore new implementations of the server.

  • Challenges:

    • What do we do with jupyter/notebook?
    • Migration pains (the APIs and traitlets bases change name).
    • Further confusion for users—where are issues, PRs, bugs, etc. coming from and where should they be reported.

• Current Progress [name=@Zsailer]

  1. Forked Notebook in July 2018

  2. Notebook + Jupyter Server in sync (Ported ~60 PRs from Notebook).

  3. Jupyter Server frontends

     • For nteract, jupyterlab, voila, notebook, etc.
     • Demo: ExtensionApp
       • Before, we subclassed NotebookApp.
       • Now, we subclass ExtensionApp.
     • Documentation: xx
     • Current examples:
       • Simple Extension Example jupyter_server#117
       • jupyter/notebook
       • voila-dashboards/voila

• [Poll] Community Adoption of Jupyter Server [name=@Zsailer]

  • Discussion topic: ask Jupyter frontends to adopt the ExtensionApp mechanism and depend on jupyter_server for the future.
  • Poll: We should encourage Jupyter frontends to adopt the ExtensionApp mechanism and depend on jupyter_server for the future.
    • Vote:
      • Endorse ++++++
      • Agree with reservation +++
      • Mixed feelings +
      • Disagree but won't block
      • Veto
    • Results and discussion.

• [Poll] The Future of jupyter/notebook. [name=@Zsailer]

  • Discussion topic: what do we do with jupyter/notebook?
    • Proposed direction 1: next major release (7.x) of jupyter/notebook depends on jupyter_server (and ExtensionApp).
    • Proposed direction 2: fork frontend code in jupyter/notebook to new repo (i.e. jupyter/nbclassic).
  • Poll: Switch jupyter/notebook to depend on jupyter_server (and ExtensionApp) with a major release (7.x).
    • Vote:
      • Endorse +++++
      • Agree with reservation ++
      • Mixed feelings ++
      • Disagree but won't block +
      • Veto
    • Results and discussion.

• Simple extension example jupyter_server#117

  • Ready for review/merge.

• Kernel Provider Mechanism

  • Jupyter Kernel Management goal is to decouple kernel from server and rely on a discovery mecanism. The added values:
    1. without changing the notebook config, have multiple providers (each responsible for the kernel lifecycle (async (re)start, terminate…).
    2. Finding the correct provider with a KernelFinder.
  • @takluyver has implemented jupyter_kernel_mgmt and jupyter_protocol repos.
  • jupyter_client is no more used (would be deprecated, to be confirmed...).
  • Various PRs last year(s) to integrate this in notebook and add async features (not merged, deprecated...).
  • Current work is in jupyter_server by @kevin-bates:
    • [WIP] Kernel providers jupyter_server#112
  • Impact on downstream projects:
    • `Classic Notebook:`` depends on previous point...
    • JupyerLab: POC to evolve jlab server components to onboard jupyter_server to be discussed on jupyterlab#7310
    • Voila: needs to adapt the exisitng voila server direct interactions with Kernels.
  • Questions:
    • Any others projects interacting with Kernels (via jupyter_client...): nbconvert...?
    • Migrate @takluvyer repos to Jupyter org?
    • Minimum python version for async support: 3.5, 3.6, 3.7?

[Zsailer]

October 17th, 2019

Attendees

Name	affiliation	username
Zach Sailer	Jupyter Cal Poly	@Zsailer
Vidar T Fauske	J.P. Morgan Chase	@vidartf
Eric Charles	Datalayer	@echarles
Kevin Bates	IBM	@kevin-bates
Luciano Resende	IBM	@lresende

There were other attendees at the meeting, but they were not added to the list.

Agenda + Minutes

• Plan to transition jupyter/notebook to depend on jupyter_server.

  • Build a notebook (jupyter) server extension.
  • Shim the notebook APIs
  • Make the PR branch into a temporary package for side-by-side testing ("nbclassic"? "nbclassic-temp"?)?
  • Release a major notebook (7.x?) release with shims + warnings
  • Release a major notebook (8.x?), dropping notebook server APIs
  • Switch JupyterLab to jupyter_server

• Jupyter Authorized Server

  • https://github.com/Zsailer/jupyter_authorized_server
  • Jupyter Server with authorization.
  • User authorization is defined by policies.

• Kernel Provider Mechanism

  • Jupyter Kernel Management goal is to decouple kernel from server and rely on a discovery mecanism. The added values:

    1. without changing the notebook config, have multiple providers (each responsible for the kernel lifecycle (async (re)start, terminate…).
    2. Finding the correct provider with a KernelFinder.

  • @takluyver has implemented jupyter_kernel_mgmt and jupyter_protocol repos.

  • jupyter_client is no more used (would be deprecated, to be confirmed...).

  • Various PRs last year(s) to integrate this in notebook and add async features (not merged, deprecated...).

  • Current work is in jupyter_server by @kevin-bates:

    • [WIP] Kernel providers jupyter_server#112

  • Impact on downstream projects:

    • `Classic Notebook:`` depends on previous point...
    • JupyerLab: POC to evolve jlab server components to onboard jupyter_server to be discussed on jupyterlab#7310
    • Voila: needs to adapt the exisitng voila server direct interactions with Kernels.

  • Questions:

    • Any others projects interacting with Kernels (via jupyter_client...): nbconvert...?
    • Migrate @takluvyer repos to Jupyter org?
    • Minimum python version for async support: 3.5, 3.6, 3.7?

  • Follow up with nb_conda_kernels

• Quick demo (5 minutes) of Classic Notebook and JupyterLab (Core) as Server Extension with Kernel Management based on:

  • notebook#4653
  • jupyter_server#112
  • jupyter_kernel_mgmt/fixup-async
  • jupyterlab/server-ext
  • jupyterlab_server/server-ext
  • PS:
    • Minor glitch for the kernel name display in classic notrebook (see this comment).
    • Goal is to have JupyterLab and Classic Notebook extensions deployed in the same Jupyter Server runtime.

• JupyterLab

  • [DISCUSS] Remove notebook dep from jupyterlab and jupyterlab_server jupyterlab#7310.
  • [NEXT MEETING] Need also to discuss on the handlers currently in jupyterlab repo (which could be in jupyterlab-server - Other option is to merge jupyterlab-server into juptyerlab).

[Zsailer]

October 24th, 2019

Attendees

Name	affiliation	username
Zach Sailer	Jupyter Cal Poly	@Zsailer
Kevin Bates	IBM	@kevin-bates
Eric Charles	Datalayer	@echarles
Afshin Darian	Two Sigma	@afshin
Steve Silvester	AWS	@blink1073
Vidar Tonaas Fauske	J.P.Morgan Chase	@vidartf

Agenda + Minutes

• [name=Zsailer] Notebook Shim

  • https://github.com/Zsailer/notebook_shim

  • Allows you to install notebook, jupyterlab, and a notebook server extension side-by-side.

  • Config priorities:

    	lab	notebook
    user	1st	2nd
    default	3rd	4th (N/A)

• [name=Zsailer] Roadmap (items are listed in order of priority).

  • Rename notebook_shim to nbclassic.
    • a notebook ExtensionApp (from jupyter_server).
    • pulls Notebook JS and templates from notebook.
    • execute from command line with jupyter nbclassic
    • migrates configuration to jupyter_server_config.
  • Update Jupyter Server JEP to reflect new nbclassic.
  • Merge Kernel Providers
    • Move jupyter_kernel_mgmt, jupyter_protocol to Jupyter's Github org.
    • Documentation for Kernel Providers, jupyter_kernel_mgmt and jupyter_protocol.
    • Merge Kernel Providers into jupyter_server: Kernel providers jupyter_server#112.
    • Help voila adjust to kernel_mgmt
      • @kevin-bates mentioned that he spoke with @SylvainCorlay about KP.
    • Release jupyter_server with Kernel Providers
    • Helpnb_conda_kernels switch to Kernel Providers.
      • They currently have an old version of Kernel Provider.
      • nb_conda_kernels tries to import Kernel Providers. If found, uses the new kernel management framework. Otherwise, falls back to old way.
  • Transition JupyterLab to Jupyter Server, dropping notebook as a dependency.
    • Uses nbclassic to run notebook and lab side-by-side.
  • Notebook 7.x: transition jupyter/notebook to Jupyter Server
    • drop the old notebook server
    • NotebookApp becomes
    • shim the notebook server API to jupyter_server
    • raise DeprecationWarnings in shimmed APIs.
    • release notebook 7.x
  • Notebook 8.x: drop shims.

• JupyterLab team meeting

  • Dec 2nd-6th, Austin, TX.
  • Push to JLab 2.0
  • JLab 2.0 won't switch to Jupyter Server
  • As soon as JLab 2.0 is out, Jlab 3.0dev should switch JS immediately
    • 6 months of testing JupyterLab on Jupyter Server.
  • Zach is planning to work on Jupyter Server, Notebook 7.0, JupyterLab-to-JS that whole week.

• Kernel Provider PR

  • Question: How does this affect custom KernelManagers, e.g. nb_conda_kernels?
    • It won't crash anything 🎉
    • KernelManagers are not configurable anymore.
    • The new kernel managment mechanism with ignore custom KernelManagers.
    • Custom KernelManagers will need transition to the new Kernel Providers framework.

[Zsailer]

October 31st, 2019

Attendees

Name	affiliation	username
Zach Sailer	Jupyter Cal Poly	@Zsailer
Kevin Bates	IBM	@kevin-bates
Eric Charles	Datalayer	@echarles

Agenda + Minutes

• [name=Zsailer] Jupyter Home
  • "Desktop" for many Jupyter Frontends.
  • Discovers Jupyter frontends and shows them in one panel.
• Bugs to fix.
  • Check that default_url routes entrypoints correctly.
  • Subcommand PR.
    • ExtensionApp doesn't currently handle subcommands.
    • Example where this is needed jupyter lab build.
  • Proposed fix: [MRG] preparse subcommands in an extensionapps jupyter_server#133
• Switch test framework to pytest.
  • Since the fork from jupyter/notebook, we've never really revisited the tests. We need to give them some attention.
  • Current test framework (nose) is brittle.
  • Pytest is more arguably readable
  • Pytest fixtures + pytest-tornado alleviate some of the issues with runnning tests locally.
  • Zach will spend this next week refactoring tests to pytest. Gives him a chance to make sure all tests make sense in jupyter_server.

[Zsailer]

November 7th, 2019

Attendees

Name	affiliation	username
Zach Sailer	Jupyter Cal Poly	@Zsailer
Kevin Bates	IBM	@kevin-bates
Luciano Resende	IBM	@lresende
Brian Granger	AWS	@ellisonbg
Eric Charles	Datalayer	@echarles

Agenda + Minutes

• [name=Zsailer] Porting Server tests to PyTest
  • Separate repo for now: https://github.com/Zsailer/jupyter_server_tests
  • Zach will finish the porting of tests and merge this week.
• [name=echarles] JupyterLab and JupyterLab Server
  • ExtensionApp into Jupyterlab Server: https://github.com/jupyterlab/jupyterlab_server/pull/79

  • JupyterLab as a Jupyter Server Extension: https://github.com/jupyterlab/jupyterlab/pull/7416

  • Example: https://github.com/jupyter/jupyter_server/pull/117

  • nbclassic needs to manage configuration migration.
  • JupyterLab will depend on nbclassic for the transition period.
• [name=ellisonbg] FastAPI/Pydantic with tornado
  • Might need to explore ASGI integration in Tornado.
• [name=echarles] jupyter kernel mgmt with jupyter server doc Enrich Docs with Usage takluyver/jupyter_kernel_mgmt#28 (look the images)
  • Both Kernel Management and Kernel Protocol have readthedocs pages up and running: https://jupyter-protocol.readthedocs.io and https://jupyter-kernel-mgmt.readthedocs.io
  • Eric made diagrams showing the Kernel Provider architecture; we should get some review over there.
  • Kernel Management and its relationship to remote terminals
    • There's work around using Kernel Gateway to launch remote terminals. It's currently working on KG.
    • Question: will the new kernel management prevent this from working in the future?
    • Terminals don't have "terminal" specs.
• [name=Zsailer] Review the RoadMap

[vidartf]

The time for the meeting seems to be different in the calendar and the README. Which one is the right one? I was there at 8 PST with @blink1073. 😕

[ellisonbg]

Zach is traveling today so he canceled - I don't remember where he posted that, but he did send out something...sorry about the miscommunication

…

On Thu, Nov 14, 2019 at 8:29 AM Vidar Tonaas Fauske < ***@***.***> wrote: The time for the meeting seems to be different in the calendar <https://jupyter.readthedocs.io/en/latest/community/content-community.html#monthly-meetings> and the README <https://github.com/jupyter/jupyter_server/blob/master/README.md#weekly-dev-meeting>. Which one is the right one? I was there at 8 PST with @blink1073 <https://github.com/blink1073>. 😕 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/jupyter/jupyter_server/issues/126?email_source=notifications&email_token=AAAGXUCC4ZU2G2S2UIIMN33QTV4HHA5CNFSM4JEGWEM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEECN35Y#issuecomment-553967095>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAGXUEOXII2QI23WF7VCV3QTV4HHANCNFSM4JEGWEMQ> .

-- Brian E. Granger Principal Technical Program Manager, AWS AI Platform (brgrange@amazon.com) On Leave - Professor of Physics and Data Science, Cal Poly @ellisonbg on GitHub

[kevin-bates]

Yeah, sorry for the inconvenience: jupyter-server/jupyter_server#111 (comment)

[vidartf]

Sure, that's ok. There's still a discrepancy between the readme and calendar though :)

[echarles]

@vidartf The README points to Thursdays at 8:00am, Pacific as written in the initial comment. When I click on the link I receive the following which I think is correct (worked for me last week).

[vidartf]

@echarles Yes, and the jupyter calendar says 7 am pacific

[echarles]

oh I see... @Zsailer can you update the jupyter calendar to 8am pacific so west coast participants have their minimal required coffee :)

[Zsailer]

Fixed :) Sorry, y'all. The discrepancy came from a recent day light savings change here.

[echarles]

Thx Zach. Searched a bit on DST Google Calendar recurring event and not clear if Google supports automatically the changes. Otherwise, someone has to schedule in his calendar to update the calendar...