Access Markdown frontmatter with env.metadata

[holmboe]

What version of myst-parser are you using?

3.0.1

What version dependencies are you using?

docutils                      0.20.1
markdown-it-py                3.0.0
mdit-py-plugins               0.4.0
Sphinx                        7.4.5

What operating system are you using?

Linux

Describe the Bug

Accessing frontmatter to use as content in the current file fails because the metadata of the file itself is not present in the env.metadata dict.

Expected Behavior

It should be possible to access env.metadata for current file as env.metadata[env.docname]["key"], just as it is possible to access env.metadata["another-file"]["key"].

sphinx-doc/sphinx#2043 suggests that the "file wide metadata" should be accessible in the env.metadata dict, and it is for all other files, but just not the metadata that is in the file at hand.

To Reproduce

A working set up to showcase the behavior can be found in this Gist: https://gist.github.com/holmboe/51dd5231301e0e282fc86922a17362ca

However, for the sake of completeness, here is a short example Markdown file:

---
last_review_date: 1970-09-08
---
The last review date of _this_ file was on {{ env.metadata[env.docname]["last_review_date"] }}

[holmboe]

It could be that #860 is also experiencing the same behavior/bug as I am.

[holmboe]

I am seeing some weird behaviour with accessing the metadata in general (outside the described issue above).

1. After a make clean and a single build the metadata is not present
2. It takes a second build to have the metadata show up

So in my case I am doing make clean; make livehtml and then making a small insignificant change in a file only to trigger a second build. After the second build the data is shown.

This could potentially be reported as a second bug somewhere (probably not in myst-parser) however that will have to be handled separately.

[holmboe]

I have tested with updated dependencies, same results:

docutils                      0.21.2
markdown-it-py                3.0.0
mdit-py-plugins               0.4.1
Sphinx                        7.4.6

[chrisjsewell]

Heya, metadata is collected in a transform, after the full document is parsed: https://github.com/sphinx-doc/sphinx/blob/05cc39d9b2a64b09504c06bfc2299aad96c85ccd/sphinx/environment/collectors/metadata.py#L28

so no you should not be relying on env.metadata during parsing (this would be the same for e.g. directives), and this is not strictly the same as the markdown front-matter.

you can already do this though with:

---
myst:
  substitutions:
    last_review_date: 1970-09-08
---
The last review date of _this_ file was on {{ last_review_date }}

[holmboe]

Thanks for the explanation @chrisjsewell! It's nice to have one way forward in the status quo. However I would like to pursue this further as I see a path forward to add custom document properties (such as "last review date" and similar) in addition to some established properties, e.g. Pandoc metadata variables and MyST frontmatter fields.

Using the substitution extension for this would allow arbitrary substitutions, however, it would force all such variables to be subkeys of the myst: substitutions: dictionary while what I would like to achive ise for the above mentioned established and custom properties to be top-level keys.

That is, I would like to access the top-level keys in one way or another.

A question to you @chrisjsewell, would it be best to raise this issue in the Sphinx project? Seeing as both sphinx-doc/sphinx#2043 that I point out and https://github.com/sphinx-doc/sphinx/blob/05cc39d9b2a64b09504c06bfc2299aad96c85ccd/sphinx/environment/collectors/metadata.py#L28 you refer to are in Sphinx itself.

[holmboe]

I just wanted to reiterate on this ticket and provide some background on the use case I am advocating for here.

As we can see in GFM YAML frontmatter

YAML frontmatter is an authoring convention popularized by Jekyll that provides a way to add metadata to pages.

Reading about Jekyll front matter custom variables we see:

You can also set your own front matter variables you can access in Liquid. For instance, if you set a variable called food, you can use that in your page:

---
food: Pizza
---

<h1>{{ page.food }}</h1>

This showcases some notable prior art of this use case.