Composing Jupytext with Jupyter Nbconvert should preserve Markdown files

[mwouts]

This problem was first revealed by jupyter-book/jupyter-book#283

There, we start from a .md file, convert it to a Jupyter Notebook, and then convert it back to a .md file using jupyter nbconvert.

Following this operation, the code cells in the Markdown document that don't have an explicit language are converted to raw cells by Jupytext, and therefore the formatting of those code extracts is lost when jupyter nbconvert converts the document back to Markdown.

[mwouts]

@choldgraf , an updated Jupytext Markdown format is available on branch 1.3.0. Now, when you open a Markdown file with Jupytext, and export it to Markdown again using nbconvert, you get something close(r) to the original.

To see exactly what I mean by 'close', you can have a look at this test:

jupytext/tests/test_jupytext_nbconvert_round_trip.py

Lines 9 to 32 in f2167f1

	def test_markdown_jupytext_nbconvert_is_identity(md_file):
	"""Test that a Markdown file, converted to a notebook, then
	exported back to Markdown with nbconvert, yields the original file"""
	with open(md_file) as fp:
	md_org = fp.read()
	nb = jupytext.reads(md_org, 'md')
	import nbconvert
	md_nbconvert, _ = nbconvert.export(nbconvert.MarkdownExporter, nb)
	# Our expectations
	md_expected = md_org.splitlines()
	# #region and #endregion comments are removed
	md_expected = [line for line in md_expected if line not in ['<!-- #region -->', '<!-- #endregion -->']]
	# language is not inserted by nbconvert
	md_expected = ['```' if line.startswith('```') else line for line in md_expected]
	# nbconvert inserts no empty line after the YAML header (which is in a Raw cell)
	md_expected = '\n'.join(md_expected).replace('---\n\n', '---\n') + '\n'
	# an extra blank line is inserted before code cells
	md_nbconvert = md_nbconvert.replace('\n\n```', '\n```')
	jupytext.compare.compare(md_nbconvert, md_expected)

You can also see the Markdown file being tested here: f2167f1 .

Please let me know if you'd like to play with that version. In that makes it simpler for you, I could publish a dev version on pypi.

[choldgraf]

Nice! It looks like in this case, code fences make the markdown file break into different cells, but those cells are still markdown cells so the formatting shouldn't change much, yeah?

This seems reasonable to me - so in this case, how would one mark in the md file that a code fence is executable?

[mwouts]

Thanks @choldgraf .

That is correct: with the implementation on 1.3.0-dev, code fences without a language, or with a language that is not understood by Jupytext, are mapped to isolated markdown cells (where the fences are preserved).

Code cells that do have a language are still mapped to executable code cells in Jupyter. Now, if you open every markdown file with 1.3.0-dev as a Notebook in Jupyter Book, you still have to decide when a particular Markdown file should be executed, or not. Maybe if it has a 'jupyter' entry in the front-matter YAML? (i.e. jupytext.notebook_metadata_filter != '-all')

[choldgraf]

Ah gotcha - thanks for the clarification.

I'm still a little iffy on whether we can assume that just because a code fence has a language attached to it, that it can be executed, however I don't think this'll be an issue in Jupyter Book. Currently I'm doing what you suggest - read in a markdown file, check if there was jupytext configuration in there. If not, then dump the notebook back to a markdown string and create a new notebook w/ one cell in it.