"pip install jupytext" should install the contents manager and the extensions

[mwouts]

Some python packages like nbdime, plotly or voila automatically deploy a jupyter configuration that activates the corresponding notebook extensions.

I'd like to do the same with Jupytext and

• its contents manager
• its jupyter notebook extension
• and the jupyterlab extension

I've started working on this on Feb 12 in branch 1.0.0. But this is not working yet.

If I run, in a conda env,

pip install jupytext==1.0.0rc4

then pip uninstall jupytext says

Uninstalling jupytext-1.0.0rc4:
  Would remove:
    c:\users\marc\anaconda3\envs\jupyterlab\etc\jupyter\jupyter_notebook_config.d\jupytext.json
    c:\users\marc\anaconda3\envs\jupyterlab\etc\jupyter\nbconfig\notebook.d\jupytext.json
    c:\users\marc\anaconda3\envs\jupyterlab\lib\site-packages\jupytext-1.0.0rc7.dist-info\*
    c:\users\marc\anaconda3\envs\jupyterlab\lib\site-packages\jupytext\*
    c:\users\marc\anaconda3\envs\jupyterlab\lib\site-packages\tests\*
    c:\users\marc\anaconda3\envs\jupyterlab\scripts\jupytext.exe
    c:\users\marc\anaconda3\envs\jupyterlab\share\jupyter\lab\extensions\jupyterlab-jupytext-0.1.0.tgz
    c:\users\marc\anaconda3\envs\jupyterlab\share\jupyter\nbextensions\jupytext\index.js

which seems to indicate that all three elements have been installed. But none of them is active in Jupyter - they only work when I manually install each of them, as documented in the README on branch 1.0.0.

[mwouts]

@martinRenou, @SylvainCorlay, it looks like your team did a similar work on voila recently. Was it easy to get the automated jupyter configuration working? Do you have any clue on what I miss here? Thanks!

[phaustin]

Not sure this is the appropriate issue for a comment on the extension -- it looks great, but I expected the "custom" item to be checked for me in the pull down menu, since I've got

c.ContentsManager.default_jupytext_formats = "ipynb,python//py:percent

Also, http://localhost:8888/tree#nbextensions_configurator is flagging the jupytext items in yellow for not showing "explicit compatibility"

Thanks again for all your work on this -- I'm giving a couple of departmental seminars at McGill next Monday and one of them will showcase working with jupytext and jupyter notebooks, so having the extension available is excellent timing.

[mwouts]

Thanks again for all your work on this -- I'm giving a couple of departmental seminars at McGill next Monday and one of them will showcase working with jupytext and jupyter notebooks, so having the extension available is excellent timing.

Much appreciated @phaustin !
And I also have to acknowledge that your feedback is another great contribution to Jupytext!

I have created two additional issues following your suggestions above, but I will certainly try to finish first the version 1.0 (won't be ready for Monday I'm afraid, as we first have to find a solution to the present issue).

Also, before officially shipping the pairing of paired notebooks into subfolders, may I ask your advice on another technical question: do you think Jupytext should create the subfolder when it does not exists, rather than fail to write the paired file?

[phaustin]

I'd be in favor of subfolder creation, perhaps with an information alert. I find that I forget folder creation about 50% of the time in new directories. Since jupyter is already silently creating .ipynb_checkpoints, and the user has installed jupytext, I don't think being proactive is a problem.

[mwouts]

@takluyver, may I ask your advice on how to best activate Jupytext's contents manager when installing the package?

I would have expected that a simple jupyter-config/jupyter_notebook_config.d/jupytext.json file with content

{
  "NotebookApp": {
    "contents_manager_class": "jupytext.TextFileContentsManager"
  }
}

would be enough, but it was not. Then I tried to replace the CM on the fly on the notebook application, but it turned out that I also had to re-run a few initialization steps (prev commit)... Is there any better solution that you would recommend?

Thanks!

[mwouts]

The re-init hack, and the automatic paired folder creations, are available in

pip install jupytext==1.0.0-rc5

I think we're coming closer, at last, to version 1.0...

[takluyver]

Sorry, I don't have an obvious answer with the config system. I would have thought that the snippet you show would work.

I'm not surprised that changing the contents manager on the fly requires a bunch of re-initialisation; most of the notebook config options only affect things at startup, and that's especially true of options ending in _class, because the notebook will create an instance as it starts and then use that.

[mwouts]

Thanks @takluyver for your answer. Now that I've seen a bit more about how the config is used, I will give a second try to the simplest approach and trace the program, and try to find out why it did not work.

[mwouts]

@takluyver , I've searched a bit more on why the simpler approach did not work. My findings are: NotebookApp.init_server_extensions load the jupyter configuration itself, and only uses the "nbserver_extensions" part, cf. here.

This is consistent with the observation that changing the contents_manager_class in the jupyter-config/jupyter_notebook_config.d/jupytext.json file does not work.

However, I was not able to find where the jupyter config is first read. Can you confirm that the part of the config that is read when the NotebookApp is created does not include the json files added by the extensions? (Is that an effect of this line ?)

[mwouts]

The recent changes (above) reduce the impact of the re-init steps. In version 1.0.1 the contents manager works with no further configuration in binder and also in a simple install of jupyterhub. Hopefully it will also solve #185 .

I have also asked for advice on this issue at jupyter/notebook#4419.

[mwouts]

The new implementation seems to work well. It has been tested: locally, with binder, with jupyterhub (with and without sudospawner).

[takluyver]

I think the configuration mechanism you were trying to use should work, but it's possible that it was being overridden by a regular config file outside the jupyter_notebook_config.d directory.

The implementation for reading this is here, with a comment explaining that config outside the directory deliberately takes precedence over files installed to the .d directory:

https://github.com/jupyter/notebook/blob/d145301b5583366fc0c5e938ded80f07a0bc1bbf/notebook/config_manager.py#L80-L104

[mwouts]

Thanks @takluyver for the link. Then, I will give another try at this!

[mwouts]

@takluyver , I did set a breakpoint in the config_manager.py, but it's not hit.

In case you want to hear more about this, this is what I find: the config is read from notebookapp.initialize through a JSONFileConfigLoader.

And that object that does not have the .d logic, so when it looks for jupyter_notebook_config.json in the etc/jupyter folder (and does not find it), it does not open jupyter_notebook_config.d/jupytext.json.

Below is a screenshot showing the call stack on the left, and the location of the code where the configuration is tentatively loaded from etc/jupyter:

[takluyver]

Gah, sorry for misleading you. You're right, it's because of the special handling in init_server_extensions; it invokes extra config machinery which isn't used for loading the rest of the config.

I hate our config system 😞

[mwouts]

No problem, anyway it was interesting to double check how this work! And thanks for taking the time to give more pointers here.