Jinja-like statements in LaTeX are interpreted

[HuangFuSL]

In LaTeX, curly braces are frequently used. But when two pairs of braces are stacked together, this plug-in will try to interpret the statement, which might cause unwanted behavior or failure. In other circumstances, one may also encounter the same problem: #78 .

The following command is used to draw a table. With the plug-in enabled, the page fails to build because {{?}} can't be understood.

\begin{tabular}{|ccc|}
    \hline
    2   & 9     & 4\\
    7   & \multicolumn{2}{c|} {\multirow{2}*{{?}}} \\
    6   &       &\\
    \hline
\end{tabular}

The documentation suggested three solutions:

1. Convert snippets to jinja2 strings. For example, {{ 2 + 2 }} is converted to {{ "{{ 2 + 2 }}" }};
2. Add {% raw %} and {% endraw %} to prevent the snippet from being interpreted;
3. Alter the syntax of jinja2 in mkdocs.yml.

In most cases, these solutions help.

However, what I want is to make the projects using mkdocs-macros and not using mkdocs-macros compatible. That is, the plug-in should be able to detect the pseudo-jinja statements and simply ignore them. The result is that if a project is originally not using mkdocs-macros, switching to mkdocs-macros will not affect the content of the pages. On the contrary, if a project is based on mkdocs-macros, the content should also stay the same except for those "real" jinja2 statements if someone removes mkdocs-macros from the project. Or, to say the least, the interpretation can be controlled outside the document.

Apparently solution 1 and solution 2 cannot satisfy the requirements. Although solution 3 does work, it does not solve the problem but hide it.

Here is some thoughts based on previous discussion. For more discussion on this problem, please refer to Issue #78.

We can intuitively assume jinja-like statements in code blocks are all pseudo-jinja statements. As a result, detecting code blocks automatically requires parsing the markdown file in the plug-in. And that means additional, possibly unworthy work.

I've came up with one possible solution (You can find it here), by adding an ignore file. But it's not sufficient for documents both including jinja statements and so-called pseudo-jinja statements. Is there a better way to resolve this problem?

[fralau]

Can we safely assume that the problem is localized in code blocks and inline code, and never happens outside of them?

[HuangFuSL]

No, as math blocks and inline math will also use LaTeX syntax. However, I did not test such cases.

For other usage scenarios, I reviewed some of the related issues:

• Issue macros plugin interprets syntax in code blocks as macro tags - fails to build #78: code blocks and inline code;
• Issue Options for custom jinja comment tag markers #70: usage of DSL;
• Issue Macros get ran while in code block #57: code blocks and inline code;
• Issue Unable to process markdown files that contain default Jinja template markers #5: looks like a code block.

I couldn't come up with any other rational circumstances where {{, {%, {#, }}, %} and #} appear except for those mentioned above. But when it comes to DSLs, anything could happen. :(

So, for cases like issue #70, my suggestion is to make a reversed approach: let users declare a marker to be interpreted rather than a marker not to be interpreted in some of the md files. Such md files need to be defined somewhere. But it might be hard to implement as the plug-in does not actually perform the processing but jinja2. With {{, {%, {#, }}, %} and #} present, jinja2 may fail to parse the document.

[fralau]

This is an excellent point, I had completely missed it.

We therefore have also math blocks and inline math.

Here would be the complete list, as far as I understand:

1. block
   • code
   • math
2. inline
   • code
   • math

Whether it's LaTeX, or some other DSL, those are the cases.

For completeness, there could also be:

1. The case where there is a competition with another templating language, which is similar but incompatible
   (as in Options for custom jinja comment tag markers #70). But in that case, I believe it would be an aporia (a logical impasse).

2. All sort of cases, where the markdown contains embedded HTML code wihich in turns contains those symbols... but that seems much less likely.

[fralau]

In general there would be two broad categories of solutions:

1. Where mkdocs-macros is told explicitly to ignore some pieces of the markdown (whether by escaping a portion of the text, or by ignoring a file altogether).
2. Where it can implicitely figure out that a piece code is actually not Jinja2.

Here are some ideas:

Category 1: Escaping

1. {%raw} and {%endraw} to bracket the part that needs to be escaped (it could be the whole markdown file). Already implemented.
2. A metadata field at the top of the markdown file. ignore_macros: true
3. An ignore list for files that the template engine should ignore (it could also have glob characters, why not).

Category 2: Detect inapplicable cases

1. Ignore code and math snippets (blocks and inline)
   1. A parameter in the yaml file (ignore_code: true)
   2. A metadata field at the top of the markdown file. ignore_code: true
2. Provide markers to be ignored (this might solve Options for custom jinja comment tag markers #70 : Tell mkdocs-macros to detect the sequence {# and } and ignore it). Unless we really want to be fancy with a parser, this is going to be crude (no embedded parentheses, strings, etc.).

Thoughts

It seems to be that category 2 is more complicated to implement, and would (by nature) not solve all cases.

I like the idea of the metadata field ignore_macros at the top of markdown file, which might be a quicky way to port a page to mkdocs-macros without breaking it. Then, if someone wanted to introduce a macro in the page, they would bracket the offending part with {%raw} and {%endraw}.

I also understand the ignore list, which could also have advantages.

Any comments, or other ideas?

[HuangFuSL]

Sorry for my late reply.

Solution 2 in category 1 can solve the problem concisely. The only problem is that when you have a relatively large number of pages to process, you need to do a lot of Ctrl-Vs (or leave the work to a script). Solution 3 in category 1 is just a complementation of solution 2. It can filter out lots of pages with the help of glob characters in a few lines. But solution 3 needs more implementation work, and things become complex when you are adding macros back to those filtered pages.

I've considered other approaches recently, but they are either hard to implement or too complicated. For what it worth, I prefer solution 2 than solution 3 in category 1.

[fralau]

That sounds good. I will then study the idea of ignore_macros at the top of the markdown file.

[fralau]

This has been implemented in commit 1f2c730. The follow up is in issue #78

[fralau]

The new doctrine, when migrating an existing project under mkdocs-macros would be:

1. Exclude the pages that are creating problems during the rendering process (ignore_macros: true)
2. This will you then leave more time to work out which of the other solutions you want to adopt for this page, in case you want to make macros co-exist with the earlier content (another solution is, of course, to change the earlier content to make it compatible 🙂).

See the new documentation.

[fralau]

@HuangFuSL Your example has been included as a test case, for the new exclusion mechanism (ignore_macros: true)

https://github.com/fralau/mkdocs_macros_plugin/blob/d458a1f4b859dd74fb8ea8421ac65a5881d4d331/test/module/docs/literal.md

\begin{tabular}{|ccc|}
    \hline
    2   & 9     & 4\\
    7   & \multicolumn{2}{c|} {\multirow{2}*{{?}}} \\
    6   &       &\\
    \hline
\end{tabular}

[fralau]

@HuangFuSL Was this discussion "answered" for you?