Improving 4.x nbextensions

[ellisonbg]

@fperez @damianavila @bollwyvl @sccolbert teoliphant @minrk @takluyver @ijstokes

I have been having a lot of on- and off-line discussion this week about the current state of nbextensions in 4.x. For a long time, we (at least this was my own logic) have hesitated to make any significant changes to how 4.x nbextensions are installed/loaded/packaged, because we know that much bigger changes are on the way in 5.0. After these recent conversations, I am convinced that we need to improve the existing 4.x nbextension architecture in the meantime. The current situation in 4.x is causing way to many problems for users and devs.

This goal of this issue is to 1) raise our community awareness that we need to do something about this and 2) come up with a concrete proposal for moving forward.

Current pain points of 4.x nbextensions

nbextensions can be installed in the following locations:

In [2]: paths.jupyter_path('nbextensions')
Out[2]: 
['/Users/bgranger/Library/Jupyter/nbextensions',
 '/Users/bgranger/anaconda/envs/python34/share/jupyter/nbextensions',
 '/usr/local/share/jupyter/nbextensions',
 '/usr/share/jupyter/nbextensions']

Config can be loaded from the following locations:

In [4]: paths.jupyter_config_path()
Out[4]: 
['/Users/bgranger/.jupyter',
 '/Users/bgranger/anaconda/envs/python34/etc/jupyter',
 '/usr/local/etc/jupyter',
 '/etc/jupyter']

By default, installed extensions are not loaded, until activated. The only place extensions can be activated is in the users config directory (~/.jupyter/nbconfig/notebook|notebook.json):

In [13]: cat /Users/bgranger/.jupyter/nbconfig/notebook.json
{
  "load_extensions": {
    "widgets/notebook/js/extension": true,
    "create_assignment/main": true,
    "nbgrader/create_assignment": true
  }
}

Pain Point #1: even though nbextensions and config can be installed in system, sys.prefix or user paths, the list of extensions to activate is only loaded from the user config. Thus, there is no way of activating nbextensions at the system or sys.prefix level.

• Example 1a: Because of this a JupyterHub deployment can't enable the various nbgrader extensions at the system level - each user has to do it.
• Example 1b: Continuum can't enable extensions on a conda env basis, even though they can install them there.

Point Point #2: There is no standard package format for nbextensions (other than a directory of stuff) and no standard way of copying an nbextension into place. Because of this, there are multiple, separate hacky ways of packaging and installing nbextensions.

• Example 2a: nbgrader has created custom subcommands, such as nbgrader extension install and nbgrader extensions activate.
• Example 2b: Continuum is starting a couple of new projects to try to solve these issues in the conda context.
• Example 2c: We ourselves, just gave up and hardcoded the ipywidgets nbextensionss in the frontend code.
• Example 2d: Projects like https://github.com/takluyver/cite2c have a separate install.py script that installs and activate its extension.
• Example 2e: @minrk recommends just using ls -n to "install" his nbextensions here: https://github.com/minrk/ipython_extensions

Proposal for addressing these pain points.

Here is the overall approach:

• Solve the above pain points in a full backwards compatible manner.
• Release the fixes quickly in a 4.2 release.
• In 5.x on the exiting pages we have (notebook/tree): formally deprecate the 4.x approach to loading/installing nbextensions, our existing JS APIs and CSS classes.
• In 5.x on the new JupyterLab page: only use new APIs and the new npm based plugin approach.

Here are the technical details of the proposal:

Pain Point 1

@takluyver has made an excellent point that the "nbconfig" frontend configuration system deliberately only loads from the users config (~/.jupyter) because these are really mean to only be "user preferences". I completely agree with this and others I have spoken to also concur. So we can't make that part of our app start to load system or sys.prefix based config.

I propose to start to load nbextension activation config from all config paths (system, sys.prefix and user) always, but do that instead by throwing that data into the page.html template itself, rather than loading later using the nbconfig web service. This would allow us to keep system data out of the user level app-preferences but still load config for nbextensions from all locations. This is not difficult to implement and is fully backwards compatible.

Pain Point 2

This one is more difficult and my proposal will be more controversial. Today, in many cases, people are shipping JS code in Python packages. For 5.0 we are going to stop doing that and embrace npm as our package format and manager, but that would require breaking changes in 4.x, so is not on the table.

I propose, that for 4.2+ we embrace putting nbextensions into Python packages:

• Don't standardize specifically where in the python package they are. This allows existing packages such as nbgrader to not have to change anything.
• Instead, 1) provide metadata in setup.py that gives the package relative paths of those assets and 2) provide that same metadata in __init__.py to enable runtime inspection. Something like this:

>>> import nbgrader
>>> nbgrader._jupyter_nbextension_paths
['nbextensions/assignment_list', 'nbextensions/create_assigment']

• Using this simple convention, we could then improve the existing jupyter nbextension command line tool to work with python packages: jupyter nbextension install nbgrader and jupyter nbextension enable nbgrader. We could also include flags the target installation/config to the system, sys.prefix and user directories. This would again be fully backwards compatible.
• Using this simple convension, Continuum can build conda installers for nbextensions with almost no effort.

[SylvainCorlay]

Another pain point, which I don't know if it could be addressed for the 4.x series is the fact that certain nbextensions depend on (a certain version) kernel-side code such as custom interactive widgets libraries.

At the moment, it is impossible to have two kernels with two different versions of ipywidgets, or bqplot, installed because they will look for the JavaScript at the same location.

Therefore, I think that the extension mechanism should acknowledge that there are two categories of extensions:

1. Global notebook application extensions, which is not dependent on kernel code (like a spell-check).
2. Kernel-dependent extensions.

A proposal to solve this would be to have the kernelspec contain one more information: a uuid which would typically be generated when the kernelspec is installed. When running a notebook with this kernel, nbextension_base_path/u-u-i-d/ would then become a search path, and the one we use for custom widgets.

cc @jdfreder

[ellisonbg]

Here is a super rough draft of a PR that addresses the first pain point: #879

Some questions that this bring up:

• Do we need separate config=True traitlets on NotebookApp for enabling common, notebook, tree and terminal nbextensions?
• What are they named?
• How do we handle multiple config files that define those? Last wins? Try to merge?
• Our .py config files allow lists to be appended/extended, but .json doesn't.
• How will folks at Continuum write these config files programatically?

There are some design decisions to make, but the good news is that it is simple code and logic to add this. Let's discuss more tomorrow.

[Carreau]

Release the fixes quickly in a 4.2 release.

I don't agree with "quickly". I would prefer "thoroughly tested" with "with good documentation and example".

Don't standardize specifically where in the python package they are. This allows existing packages such as nbgrader to not have to change anything.

I would still like if you make extensions in python packages to be able to activate them per submodule: jupyter activate nbgrader.student / jupyter activate nbgrader.teacher

throwing that data into the page.html template itself

Does that implicate you need to restart the server on loading extensions ?

[Carreau]

Extra note, if the packages are Python and have versions, we should be able to
serve multiple versions at differents url like /nbextension/<extension>/<version>/, if version is omitted latest is implied.

[minrk]

Re: @SylvainCorlay's point, also during that discussion, we proposed a kernel-specific nbextension path, which we never got around to implementing. I feel bad about that. I think we ended up proposing an nbextensions dir inside the kernelspec that kernel-specific extensions could go in.

It seems a bit worrisome that we would officially bless Python packages as the temporary solution for 4.x nbextensions, and then turn 180º and say that it's all npm, and not Python packages as fast as we can. But if we want to make it more convenient, adding a flag for 'install js from a Python package':

jupyter nbextension install --py nbgrader

seems like a better middle ground than breaking jupyter nbextension install [path], or guessing whether [path] is a path or a package.

Do we need separate config=True traitlets on NotebookApp for enabling common, notebook, tree and terminal nbextensions?

I don't think so. What would these common nbextensions be?

How do we handle multiple config files that define those? Last wins? Try to merge?

I would do it the same way we do with the rest of config, where the more specific config has priority: user > env > system. I think that's the only thing that's missing from nbconfig for this to work.

How will folks at Continuum write these config files programmatically?

During the nbextension discussion a year ago, it was concluded that we must not activate extensions as part of installing them - that install & activate must be two separate actions. Are we changing our minds on that?

If someone wants to enable nbextensions via conda packages, the biggest hurdle is that all enabling currently resides in a single file, and conda packages should only write, not modify files. To support this, the only idea I have is a config.d-style directory of config files, so that packages could drop a file in there, and all such files are loaded.

[Carreau]

Re: @SylvainCorlay's point, also during that discussion, we proposed a kernel-specific nbextension path, which we never got around to implementing.

You can load extension from kernelspec directory using kernel.js, you just need to do relative requires.

It seems a bit worrisome that we would officially bless Python packages as the temporary solution for 4.x nbextensions, and then turn 180º and say that it's all npm, and not Python packages as fast as we can. But if we want to make it more convenient, adding a flag for 'install js from a Python package':

What about "just" adding npm global dir to the search path ?

I would do it the same way we do with the rest of config, where the more specific config has priority: user > env > system. I think that's the only thing that's missing from nbconfig for this to work.

For general frontend config, it's hard to merge JSON, though for list of extension, that might be easier.

[Carreau]

that install & activate must be two separate actions. Are we changing our minds on that?

I don't think so, I think the setup.py list is to discover extensions.

[jankatins]

Just that I understand it correctly:

This:

embrace npm as our package format and manager

just means that developers of packages need to use npm, not that users of the package need to install npm in addition to python+python package manager?

If not: I don't think it's good for the adoption of extensions if e.g. a julia user has to install a python environment to get a python package manager to install a jupyter notebook and then start installing node and npm to get extensions. Learning python (the language, the tools, the libs...) instead of using a preinstalled SPSS/STATA/... is already hard for students and researchers, so don't add on learning another packaging ecosystem to get notebook extensions.

[parente]

Maybe this belongs in a separate issue, but it seems like the right time to fix it along with these other problems. Installing server-side extensions suffers from a similar pain point about jupyter_notebook_config.py vs jupyter_notebook_config.json configs.

We found out the hard way that the JSON config takes precedence over the Python config in jupyter/dashboards#153. So if some server-side extensions ask users to add themselves to the .py config like c.NotebookApp.server_extensions = ['my.extension'] (currently: nbexamples, dashboards, declarativewidgets) while others use the ConfigManager class to add themselves to the .json config (nbgrader, anything using the nbsetuptools), only the .json ones will wind up getting enabled.

I have no problem switching over to one or the other, but, which is the correct one? Or does something need to change so that both are supported and the lists of extensions are merged across the config types?

[minrk]

I have no problem switching over to one or the other, but, which is the correct one?

It's probably correct for .py config files to have higher priority, since they are generally human-edited and more powerful, while .json config files should always and only be programmatically edited. .py files can be considered 'manual overrides'.

In Python, it makes sense to do c.NotebookApp.server_extensions.append('my.extension'). The JSON config files cannot do this, since they are a simple dict-dump. And further, they don't generally need to, because opening and appending to a list in a JSON file is doable, unlike in Python. A further point in favor of .py files having higher priority.

[Carreau]

In Python, it makes sense to do c.NotebookApp.server_extensions.append('my.extension'). The JSON config files cannot do this, since they are a simple dict-dump. And further, they don't generally need to, because opening and appending to a list in a JSON file is doable, unlike in Python. A further point in favor of .py files having higher priority.

We already had a long discussion on which one between py an json should take precedence and decided on Json, as we want at some point to edit configuration only through UI. Having .py taking precedence would mean in many cases that if a user does: c.NotebookApp.server_extensions = ['my.extension'] no automatic tool can ever activate an extension. So which one takes precedence is still not obvious to me, and I'm not sure a single case like this one change the balance.

Json could perfectly have append also, as long as we decide that an append or prepend keys become a LazyAppend/LazyPrepend.

We maybe should just warn more loudly if one file erase the settings of another, or put some time in looking at tools like redbaron to modify a Python config file in obvious case when a config option is not dynamic.

[parente]

We maybe should just warn more loudly if one file erase the settings of another ...

The warning is definitely there and pretty loud on server start:

[W 14:29:11.533 NotebookApp] Unrecognized JSON config file version, assuming version 1
[W 14:29:11.536 NotebookApp] Collisions detected in jupyter_notebook_config.py and jupyter_notebook_config.json config files. jupyter_notebook_config.json has higher priority: {
      "NotebookApp": {
        "server_extensions": "<traitlets.config.loader.LazyConfigValue object at 0x7f15cc47dc18> ignored, using ['urth.dashboard.nbexts']"
      }
    }

But I'll admit @jtyberg and I missed it for some time when debugging the issue I linked above. Even when we did see it, we only knew what to do because we're expert-amateurs in how the Jupyter config system works. I'd bet a typical notebook user wouldn't have an easy time rectifying the problem.

Json could perfectly have append also, as long as we decide that an append or prepend keys become a LazyAppend/LazyPrepend.

If you wanted to scope it down to prepending/appending sequence values instead of having to merge arbitrary config objects, that would still solve the extension problem specifically without growing into a general purpose "merge all the configs!" solution.

[ijstokes]

What is the non-Continuum-camp feeling about the importance of being able to support "sandboxing" of extensions? For us (at Continuum) that means having a clean mechanism to be able to use conda environments to sandbox different sets of extensions and perhaps different versions of the same extension. From our work which has mixed both JS and Python code in the extensions conda packaging and conda environments have worked really well.

I'm not clear on how this works in either a pure-Python-package or pure-NPM-package world. If there is a good and clean solution that doesn't include conda, great. But is there scope to discuss how conda can fill the need here to manage cross-language packaging, or has that ship already sailed?