freeze_metadata and ipynb files

[kiendang]

My understanding is that

c.ContentsManager.freeze_metadata = True

is equivalent to

c.ContentsManager.default_notebook_metadata_filter = "-all"
c.ContentsManager.default_cell_metadata_filter = "-all"

However, when using the later, when create a new ipynb file, metadata_filter is set

{
  "jupytext": {
    "metadata_filter": {
      "notebook": {
        "excluded": "all"
      },
      "cells": {
        "excluded": "all"
      }
    }
  },
  "kernelspec": {
    "name": "python3",
    "display_name": "Python 3",
    "language": "python"
  },
  "language_info": {
    "name": "python",
    "version": "3.6.5",
    "mimetype": "text/x-python",
    "codemirror_mode": {
      "name": "ipython",
      "version": 3
    },
    "pygments_lexer": "ipython3",
    "nbconvert_exporter": "python",
    "file_extension": ".py"
  }
}

while if using the former, metadata_filter is not set

{
  "jupytext": {},
  "kernelspec": {
    "name": "python3",
    "display_name": "Python 3",
    "language": "python"
  },
  "language_info": {
    "name": "python",
    "version": "3.6.5",
    "mimetype": "text/x-python",
    "codemirror_mode": {
      "name": "ipython",
      "version": 3
    },
    "pygments_lexer": "ipython3",
    "nbconvert_exporter": "python",
    "file_extension": ".py"
  }
}

This leads to some confusing behavior when setting c.ContentsManager.freeze_metadata = True and creating a paired notebook from a ipynb file.
i.e If I set c.ContentsManager.freeze_metadata = True then create a .ipynb file, e.g with 1 cell import functools, then create a paired notebook by adding "formats": "ipynb,py" I would expect the corresponding .py file would be just

import functools

instead I get

# ---
# jupyter:
#   jupytext:
#     formats: ipynb,py:light
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.3'
#       jupytext_version: 0.8.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
#   language_info:
#     codemirror_mode:
#       name: ipython
#       version: 3
#     file_extension: .py
#     mimetype: text/x-python
#     name: python
#     nbconvert_exporter: python
#     pygments_lexer: ipython3
#     version: 3.6.5
# ---

import functools

This does not happen if I use

c.ContentsManager.default_notebook_metadata_filter = "-all"
c.ContentsManager.default_cell_metadata_filter = "-all"

or create the py file first.

[mwouts]

Hello @kiendang , thanks for reporting this.

With Jupytext 0.8.4 (current version) if you do not want any metadata in your scripts, you should indeed use "-all" for the default filters, as in your example above.

The purpose of the freeze_metadata option is a bit different: it sets the metadata filter to match the actual metadata of the text file. This is useful if you want to preserve some metadata there (think of R Markdown files for instance).

The freeze_metadata option was recently introduced, and we can certainly improve it. The case you report - a text file that did not exist before - is interesting. What should freeze metadata mean in that context? Either freeze to empty, as you would have expected, or freeze to ipynb content, as is happening?

By the way: I note that the implementation currently overrides the jupytext metadata itself. That is a bit to much: instead, it should only "update" the corresponding parts of the dictionary. I will have to make sure that even if the jupytext metadata is filtered, the format information is still preserved (test case: open a '# %%' script with the freeze_metadata option, save it, and test that format is preserved).

[mwouts]

Hello @kiendang , I have updated the documentation for the next release. Do you think the new section on configuration , where I explain the difference between default_notebook_metadata_filter and freeze_metadata, is clear enough?

[kiendang]

Thanks for taking the time to do this.
For me, reading the doc, I still can't really tell the difference between

If you want that the generated text files have no metadata

c.ContentsManager.default_notebook_metadata_filter = "-all"
c.ContentsManager.default_cell_metadata_filter = "-all"

and

you don't want Jupyter to add a YAML header
Do not add metadata to existing scripts

c.ContentsManager.freeze_metadata = True

Does it mean that when I use a script which already has some metadata, if I use the former, those metadata will be cleared out, but if I use the later, they will be preserved, and no other metadata will be added? And if I create a new R or python text file, the results would be the same for either approach?

[mwouts]

Thanks for taking the time to do this.

You're welcome! Thanks for reviewing the doc, and for your feedback.

Does it mean that when I use a script which already has some metadata, if I use the former, those metadata will be cleared out, but if I use the later, they will be preserved, and no other metadata will be added?

Yes, it does!

And if I create a new R or python text file, the results would be the same for either approach?

It depends:

• No, if you create it by pairing a script to a Jupyter notebook,
• Yes, if you create the script outside of Jupyter, and then edit it in Jupyter.

The reason why I introduced the freeze_metadata is that pre-existing scripts vs notebooks are two different use cases. Personally I want to have the metadata if the script represents a notebook, otherwise, just like you, I do not always appreciate the extra YAML header in scripts.

Can you tell me which config you use, in the end? Probably just the metadata filter?

One more question: do you think the freeze_metadata=True option could become a default behavior of Jupytext in Jupyter: unless the users choses to turn the script into a notebook, Jupytext should not add any metadata to it?

[mwouts]

(sorry the close was not intentional)

[kiendang]

(sorry the close was not intentional)

github automation's gone too far :))

Thanks for explaining how freeze_metadata works to me. I went and experimented a bit and I think I understand now.

I submitted a pull request modifying the doc a bit. I think it's useful to keep the example of hand-picked metadata filtering. Also I think describing freeze_metadata as preserving metadata in pre-existing text or markdown files is clearer, at least to me.

Let me know what you think.

[kiendang]

Can you tell me which config you use, in the end? Probably just the metadata filter?

For my original use case (I don't want to keep any metadata) I use default_notebook_metadata_filter and default_notebook_metadata_filter. I wasn't fully aware of what freeze_metadata does. I agree that these solve two different problems. I would totally support setting freeze_metadata=True as default. This would be useful in cases when other people sent me text files created by Jupytext with some metadata. If I use Jupytext to open those files I would want to keep those metadata as is.

[mwouts]

Thanks @kiendang for your feedback, and also for the PR. I will review it and integrate it tonight.

I would totally support setting freeze_metadata=True as default.

Good! I think I also would. I will set that option for my own use and test, and if there is not drawback of doing so, I may integrate that in the next release.

[mwouts]

Hello again @kiendang , can you tell me if the RC solves the issue? Note that you will have to surppress the freeze_metadata option. Thanks

[mwouts]

I expect this to be fixed in the RC (now at 1.0.0-rc2):

pip install jupytext --upgrade --pre

@kiendang, please reopen if you see any issue. Thanks.