opening jupytext notebooks in jlab

[parmentelat]

Hiya

trying to migrate from classic to something more modern, I'm facing an issue that has to do with the awkward path that seems to be the only way to open a jupytext notebook under jlab

• from the UI, clicking say foo.md in the file browser results in a markdown editor; I need to do right-click → open with → Notebook to get it to open as a notebook
• from the outside, I'd need a way to do the same from a regular URL; opening lab/tree/path/foo.md has the same result as from the UI, so I can't seem to find any way to open it as a notebook from a URL

the first issue is an impediment but of course something I can live with; the second is more of a showstopper for me at this point

I'd love to find a way around this

ps
for reference, I've be stirring this in a couple other threads already
https://discourse.jupyter.org/c/jupyterlab/17
and more marginally in
jupyterlab/retrolab#162

in the latter @jtpio suggested that an extension could come in handy, but I must honestly admit that the whole jlab extension ecosystem has scared the hell out of me a bit, so I am unsure what this would actually involve..

[mwouts]

Hi @parmentelat , well these two issues have been there for a while, cf #271 for instance. But sure I'll be happy to give another try to get them fixed.

For the first point, i.e. the default viewer for .md files, could you give a try at the Jupyter Lab user settings as documented here? Please let us know if that works for you.

I understand that the second item is a bigger problem. I remember discussing with @jasongrout how to change the default viewer for the .md links. That involved overriding some "open" function, but that was a while ago and I can't remember which function that was exactly, and I am not sure I know how to do that in the Jupytext extension neither... help is welcome!

[joelostblom]

For me, changing the document user settings does not seem to change the default viewer for .md files, so I opened jupyterlab/jupyterlab#11195

[alwaysmpe]

Was wanting to set the notebook viewer as the default for python also. Having poked around in the jupyterlab code, looks like there are a few ways forward.

The problem is that jupytext isn't registering as a widget factory and the notebook viewer doesn't explicitly support python/markdown/whatever files (my understanding of jupyter/jupytext internals is pretty rough, but I am a software engineer).

Looks like when a viewer is configured in the setting, it's then trying to map that to the viewer.

This is the CR that added the ability to specify a widget to viewer mapping:
jupyterlab/jupyterlab#6813

Using this registry of viewer factories:
https://github.com/jupyterlab/jupyterlab/tree/997f3af68396539136ab0cb8512c73c07e021e4a/packages/docregistry

The Notebook viewer registers a limited number of file extensions (and I think mime types). It's not possible to add a new type to an existing viewer (ie add python to the notebook viewer) according to this issue:
jupyterlab/jupyterlab#7099

That issue proposes mapping mime types to viewers, also not currently done.

The way I see it, there are a few options to enable the desired behavior.

1. Implement the changes proposed in the above issue (there's a partial PR, I haven't looked at what progress was made), then in jupytext make the necessary changes to add the notebook mime type to python files
2. Wrap the notebook viewer, register jupytext as a distinct viewer, explicitly support whichever file formats desired, perform whatever internal changes are needed.

Option 2 would probably be quickest, but a bit of a hack. Option 1 would likely be preferable long term.

The code for the notebook viewer (as an extension) is this I believe: https://github.com/jupyterlab/jupyterlab/tree/997f3af68396539136ab0cb8512c73c07e021e4a/packages/notebook-extension

And I think it's the widget factory stuff that would need modifying, ie use a different factory name.

That's as far as I got. I'll probably look a bit more at some point, but I'm a little busy. Any pointers welcome.

[alwaysmpe]

This is confusing, so I figured I'd gather a few links to other discussions in one place.

Looks like there's a difference between how links are handled and opening from a file browser (I think, need to read a bit more of this)
jupyterlab/jupyterlab#7776

A fairly equivalent discussion for markdown
#271

This is a bit of a bodge that adds the option to the 'open with' menu, a similar hack could probably also be added when setting defaults (not ideal): https://github.com/jupyterlab/jupyterlab/pull/5247/files

[mwouts]

Thanks to @fcollonval 's contribution #877 we can now:

• open text notebooks with a single click in Jupyter Lab when "Jupytext Notebook" is configured as the default viewer for these notebooks, see the documentation
• open text notebooks with the Notebook view on Binder (requires a default_setting_overrides.json file), like e.g. this one: https://mybinder.org/v2/gh/mwouts/jupytext/main?urlpath=lab/tree/README.md

This new option will be made available with Jupytext 1.13.2 (soon on pip), so I will close this issue (comments are welcome and a confirmation that this also works for people here would be great!)

Note: the question of opening links to .md files as notebooks is still open at the moment, and will be treated separately at #271 .

[astrojuanlu]

Trying to reproduce this for some MyST notebooks, but I can't seem to be able to make it work poliastro/poliastro#1410

I changed "jupytext-md": "Jupytext Notebook" by "jupytext-myst": "Jupytext Notebook", neither option worked. At least I checked that the settings are being properly imported to the Document Manager. Any help welcome 🙏🏽

[fcollonval]

@mwouts from the comment of @astrojuanlu I realize that Jupytext is handling on the server side lots of file extensions than the one appearing for now in the labextension (for instance the one of MyST notebooks). To support them with the one-click feature, they need to be listed as supported extension by the new file type jupytext-md at

jupytext/packages/labextension/src/index.ts

Line 431 in 6fee30c

extensions: [".rmd", ".qmd"],

I let you (or someone with a better knowledge of the supported format) update that list as needed to match the supported extensions.

[mwouts]

I see! Sure I'll do that, I will try to find some time to fix that tonight... Thanks for the pointers!

[mwouts]

Well... @astrojuanlu , which extension do you use for the MyST format?

If you use the .md extension then this user settings should do the trick:

{
  "@jupyterlab/docmanager-extension:plugin": {
    "defaultViewers": {
      "markdown": "Jupytext Notebook"
    }
  }
}

At least it works on Binder with the current version, see this example: https://mybinder.org/v2/gh/mwouts/jupytext/main?urlpath=lab/tree/demo/World population.myst.md

Of course I can add the .myst, .mystnb, and .mnb extensions to the "jupytext-md" file type, but before that maybe I'd prefer to get a confirmation that someone really use them 😄 ?

[astrojuanlu]

I'm currently using .mystnb. The reason is that I think MyST-Parser is registered to parse .md files, and I found some sort of conflict. But honestly I would love to switch to .md because then they would have a nice preview on GitHub. MyST-Parser and MyST-NB were in its infancy when I did the migration and... well, I've become an expert in Sphinx and documentation this year 😃 so I might give it a try again.

[mwouts]

Thanks @astrojuanlu for the details, now I'm sure there is at least one user for them so I am willing to add these extensions to the file type 😄

But maybe that is also the opportunity to create a few more file types?

@chrisjsewell , @jjallaire, what would you think of replacing

jupytext/packages/labextension/src/index.ts

Lines 426 to 433 in 002fa06

	// Define file types
	app.docRegistry.addFileType({
	name: "jupytext-md",
	displayName: trans.__("Markdown File"),
	// Extension file are transformed to lower case...
	extensions: [".rmd", ".qmd"],
	icon: markdownIcon
	});

with three file types, as follows:

• a file type "R Markdown" for the .Rmd extension
• Another file type "Quarto" for the .qmd extension
• And finally a "MyST Markdown" file type for the .myst, .mystnb and .mnb extensions?

(@fcollonval , are we allowed to use spaces in file types?)

[jjallaire]

Yes, that sounds good to me!

[fcollonval]

are we allowed to use spaces in file types?

I don't know of any example. But I think it is find.

One comment of creating multiple file types, a file type determines what viewer can be used for it - and some styling in the file browser (like the file icons - that get override by Jupytext anyway). So I would recommend being careful about adding many of them, especially if a classical pattern for a user is then to override the default viewer for all of them.

But this is possible, see the image viewer for example:

https://github.com/jupyterlab/jupyterlab/blob/967f478b591a2bf4f569994e60fa2f168b9b4fff/packages/imageviewer-extension/src/index.ts#L116-L123

[chrisjsewell]

thanks @fcollonval!
Quick question (to lazy to read the docs right now lol), if you have a .md file, with the top-matter specifying myst as the format, will this auto-open it as MyST-Markdown?

[fcollonval]

If the default viewer is then changed to the new "Jupytext Notebook", yes it will directly open with the notebook viewer.

By default the behavior of opening them with the text editor is kept.

[mwouts]

I am experimenting with something like this: 73ddc83

However this only works for the file types that are lower case ("markdown", "python" and "julia"):

So I'm going to try with lower case file types now 😄

[mwouts]

With b8ee502 it works much better:

I also note that opening the .mystnb file failed (because it was empty? I'll open an issue for that). And the .qmd file failed to open as a notebook on Binder because we don't have quarto installed there yet.

[mwouts]

A quick wrap-up... I have a PR at #882 that seems to work well.

The new settings look like this:

{
  "defaultViewers": {
    "markdown": "Jupytext Notebook",
    "myst": "Jupytext Notebook",
    "r-markdown": "Jupytext Notebook",
    "quarto": "Jupytext Notebook",
    "julia": "Jupytext Notebook",
    "python": "Jupytext Notebook",
    "r": "Jupytext Notebook"
  }
}

and the users can choose the subset that matches their preference. The "markdown" file type covers the .md extension and "myst" covers the non .md MyST Markdown extensions.

@fcollonval , I have seen your advice about not creating too many file types - I believe this setting is a bit more appropriate because we're talking of different document types here (and even different user clusters probably). Also I have added the "julia" and "R" file types in this setting example for non-Python notebook users (I'm not sure about the other languages - do they already have a file type?)

@chrisjsewell , @jjallaire , in the end I had to change the file types names (lower case, no space), are you OK with the new names: "myst", "r-markdown" and "quarto" ?

[jjallaire]

Yes lower-case fine by me!

[chrisjsewell]

yeh no worries

[mwouts]

The new version jupytext==1.13.3 is available on pypi.

@astrojuanlu , @parmentelat , please let us know if it works according to your expectations !

Please remember that you need to override the default settings for the "Document Manager" section with (a subset) of

{
  "defaultViewers": {
    "markdown": "Jupytext Notebook",
    "myst": "Jupytext Notebook",
    "r-markdown": "Jupytext Notebook",
    "quarto": "Jupytext Notebook",
    "julia": "Jupytext Notebook",
    "python": "Jupytext Notebook",
    "r": "Jupytext Notebook"
  }
}

You can also use a local default_setting_overrides.json file as we do on Binder, maybe we could add a mention of that in the documentation...

[parmentelat]

Hi @mwouts

I have

• installed the latest jupytext 1.13.3
• and added this config file

  cat ~/.jupyter/labconfig/default_setting_overrides.json
  {
    "@jupyterlab/docmanager-extension:plugin": {
      "defaultViewers": {
        "markdown": "Jupytext Notebook",
        "myst": "Jupytext Notebook",
        "r-markdown": "Jupytext Notebook",
        "quarto": "Jupytext Notebook",
        "julia": "Jupytext Notebook",
        "python": "Jupytext Notebook",
        "r": "Jupytext Notebook"
      }
    }
  }
  

with that in place

• if I click on a jupytext file in the file browser it does open a notebook
• and I can also open a notebook from the outside by instructing my browser to open a URL like e.g.

  http://localhost:8888/lab/tree/path/to/notebook.md 
  

so yes, it's definitely much better ! and many thanks @mwouts for all the hard work :)

---

this will indeed help moving infrastructure tools towards jlab

but I just wanted to add however that it's a little bit of a pity that an extra configuration step is required here
in particular wrt beginner students
having them do that sort of (for them) black magic is not s.t. I'm going to try and do ;)
I guess they're going to have to use the right click → open with... thingy for another while
and it's a pity also because I find them more and more executing their notebooks under vscode directly, that as far as py:percent is concerned, is soooo much easier to get started with...

[mwouts]

so yes, it's definitely much better !

Thank you @parmentelat ! And actually the credit is for @fcollonval who unblocked this subject, and also addressed the other one about links in notebooks (#271, should be fixed with Jupyter Lab 4.0)

but I just wanted to add however that it's a little bit of a pity that an extra configuration step is required here

Sure I understand. We chatted about that with @fcollonval , and I was the one who asked for not changing the default configuration, at least for now, because I did not want to surprise all users with this. My experience with Jupyter Notebook was that a few people did not like the new default for .py files, so I think it is important to be able to configure that settings.

Maybe we can consider changing the default in a future (non-minor) version of Jupytext, or even maybe another option would be to create another package that will just install this configuration file? What do you think?

[fcollonval]

another option would be to create another package that will just install this configuration file?

As the override json file could be set/modify by the user or an system administrator. I advice not going that path. One possibility would be to provide in the other package, a tiny JupyterLab extension that override the default factory. Or to prompt the user if he wants to change the default viewer if he used the open with editor.

[astrojuanlu]

The new config does work as expected, thanks a lot!

[parmentelat]

@mwouts

Maybe we can consider changing the default in a future (non-minor) version of Jupytext, or even maybe another option would be to create another package that will just install this configuration file? What do you think?

@fcollonval seems to warn us against that idea; I don't care much about how exactly this happens, what really matters to me is that

• ideally no extra installation step are required
• but if that's not possible, then it would help if that extra step could involve one single terminal command, and I have to admit that a pip install something would be nice; this way I could just augment my requirements.txt and all would be just fine

now again, all this is to help us lead students towards using jupyter lab rather than jupyter notebook :)

[fcollonval]

another option would be to create another package that will just install this configuration file?

Good news, with the newly released JupyterLab server 2.9.0 this could be a thing using the new overrides.d folder.