Custom environment using Fenced Divs

[cderv]

This will resolve #924

The aim is to use fenced Divs to create the custom block, instead of using special knitr's engines. Some notes in this PR

Engines

• eng_theorem and eng_proof are in bookdown
• they are calling knitr:::eng_block2

Tricky stuffs

• \BeginKnitrBlock are used but we may not need them anymore
• A special option is supported ("bookdown.output.markdown") to output as md... not sure we want to keep doing this.
• For latex, some stuff happens in restore_block2() - we should keep some
• Reference are dealt in bookdown base on a syntax (#type:id) - that means we need a unique id, even if not is provided. Currently chunk name is used but with fenced divs, pandoc can't access that.

• Check internationalization : https://bookdown.org/yihui/bookdown/internationalization.html ?
  • For PDF it happens in restore_block2()
  • For HTML it happens in parse_fig_labels run in resolve_refs_html

Conflicts

• See how this should work regarding latex-divs.lua in Rmarkdown. There is conflict.

Current status

Something like this should work for now in this draft PR. It is not perfect but it is a start.

---
title: "test2"
output: 
  bookdown::pdf_document2:
    keep_tex: true
    pandoc_args: !expr sprintf("--lua-filter=%s", here::here("inst/resources/lua/custom-environment.lua"))
  bookdown::html_document2:
    keep_md: true
    self_contained: false
    pandoc_args: !expr sprintf("--lua-filter=%s", here::here("inst/resources/lua/custom-environment.lua"))
---

```{theorem, pyth, name="Pythagorean theorem"}
For a right triangle, if $c$ denotes the length of the hypotenuse
and $a$ and $b$ denote the lengths of the other two sides, we have

$$a^2 + b^2 = c^2$$
```

It could be referenced: see \@ref(thm:pyth)

:::{.lemma #first-lem data-name="My first Lemma"}
Here is another, containing some R code
```{r}
1+1
```

and some markdown **bold text** with an inline `r "code"` and some equation on how
you calculate $x$ 

$$
x = y + z^2
$$ 

:::

Make sure that it could be reference \@ref(lem:first-lem)

There are a lot of things to check because of how things works currently:

I'll keep working on this to make it work, and we can discuss the different choices here

Must do

• Add tests - new testthat snapshoting may help here This will be a future project to use expect_snapshot
• Make it option so that knitr engines still works - would be Pandoc Divs or knitr engines
• Add a switch in rmarkdown do deactivate a lua filter or choose the execution order (solving the conflict between both filter)

[tchevri]

This is great, very exciting - I'll be happy to try it out whenever you think I won't be wasting your time, that's probably the most I can offer as clearly I don't play in your court when it comes to coding. This said, I don't know if it's relevant, but my bookdown creates also slides, as per a previous discussion. I noticed that when I have incremental bullets, it screws up the numbering! So on the first slide, a theorem may appear as theorem 3.1, but if the slide is repeated because of an incremental bullet, then this theorem incorrectly changes to 3.2 though it's the exact same theorem, not a desirable outcome imho. I thought i'd highlight it, as I am sure it will be an easy fix for you, but not something you might have necessarily spontaneously thought about.
Again, many thanks for your awesome progress, again, very exciting for all of us to see this feature finally happening.

[cderv]

Slides output are not really an obvious bookdown output, so yes I did not thought of that. I'll have a look at your example.
But does it works correctly with the already existing theorem knitr engine ? Or is it more a feature request ?

Currently I am trying to reproduce what exists, but we can surely make improvement to make this more generic.

[tchevri]

True, but isn't it desirable to have some slides with your book?
In particular if you want a textbook?
Also, beamer / beamerarticle made the combination pretty smooth - so i naively thought this could be kept, but i was wrong, i think.
However, XieYiHui helped me with creating slides as a bookdown output, it almost works perfect, just a few issues.
A bit of a hack of your topic here, I apologize, but as an fyi, issues remaining are

1. Can't seem to be able to assign an output filename on the fly, this has been asked by others too, see comments.
2. There was a nasty pandoc "upgrade" that unexpectedly threw my code off and now forces me to stick with pandoc 2.6. I have received a few answers from John McFarlane, but nothing helpful that I know of unfortunately - it's been over a year.
3. Custom environments, that's your current post.

So yes, right now everything works perfect for knitr engine - except for the ability to put code insides the ```{} ```, and that kind defeats the purpose, because the (my) goal is to have conditional content, hence the need for executable code to allow the switch (e.g. `r if (out_type=="beamer") "-"`). With regard to the current knitr engine, there is only this one issue that I ran into when inside the theorem knitr engine, I had incremental slides and saw that as the same slide was shown with incremental material, as I said, but the theorem numbering would also be incremented, which is clearly not desirable.

again - here is my opportunity to thank you very warmly, as i have very much benefited from all your work and help! Merci infiniment.

[cderv]

Thanks for the feedbacks. I'll take that into account. For new features, don't hesitate to open new issue if none already is. IMO It is easier to follow up on one feature per issue, than one topic issue with several bugs and feature request inside it.

[cderv]

Two of the issues has been solved now.

About the identifier, we add one for eng_theorem in order to reference but not with eng_proof. Maybe we can add and if to both but on the div instead of the span ? 🤔

[cderv]

For easier testing, I added the lua filter by default.

One think I noticed: Currently we don't have good helper function in rmarkdown so that lua filters can be used in other package. I needed to add a function in bookdown, similar to rmarkdown:::pandoc_lua_filters. I think we should consider to improve this in rmarkdown so that it can be used in other 📦

Anyway, we need now to decide when and where this will work:

• Do we make this an opt-in mechanism ? Or an opt-out ? No mechanism - we make it the default
• Should it be with an option or with an argument passed to the format function ?
• Do we want to support both eng_proof and eng_theorem with these new custom Divs ?
  • If so, then we need some more check.
  • If not we may need a fail safe to detect both are in used and opt-out for example ➡️ if simple otherwise only document and offer a function to translate engine block to pandoc divs

Todos

• Needs to figure out what to do with latex-divs.lua filter conflict. Maybe changing the order of the filter could help. 🤔 Add lua_filters to pandoc_options to expose them in the output_format rmarkdown#1899 will help
• Add support for markdown_document2 family too
• Add a small helper to translate engines to Fenced div for bookdown project and document (could be done after this PR)
• Tests more
• Document this new feature
• Add this in bookdown_demo (as it will help testing)

[turtlegraphics]

Hi! I'm currently writing a stat textbook with Bookdown. We've got a mid-October deadline and realized a couple of weeks ago that fenced divs would solve all of our problems with exercises, solutions, and theorem style numbering. I wrote a lua filter to replace latex-div.lua that implements fenced divs in both HTML and LaTeX. It's probably not exactly what's needed for integration with the bookdown package, but we are using it now for a 400+ page book and it's working well.

It was a snap to handle the LaTeX side. The HTML side was much harder, because the cross references needed to be done by hand. I've dealt with a lot of issues and pitfalls and maybe can be of service.

I've got a test book I'm using to build out technologies that our real book will use (like a single-chapter PDF makefile). It's got the lua filter, fenced-blocks.lua, and you can see it in action by building the book. You need to disable the built-in latex-div.lua, which I've done by simply commenting out its contents on my system.

I'm in the thick of the Fall semester and also trying to hand our book to CRC by mid-October, but I'm happy to help where I can. You're welcome to use the code however you like.

[cderv]

Hi @turtlegraphics !

Thanks a lot for sharing your work ! This is really awesome ! There are a lot of great ideas !

I really like the numbering and cross referencing from the lua filter - we were not planning to do that from the lua filter but as you managed to do it, it could be interesting to add. Also the special classes to allow further customisation in HTML is really cool !

We may integrate some of this and also fix some of your pitfall (latex-divs.lua conflicts, latex preamble or i18n). If I have question I'll ping you, but in the mean time it would be awesome if you find time to contribute here to make all this supported in bookdown (by testing or just by telling me which are the important feature to have so that I don't forget).

[cderv]

To remove after merging the PR in rmarkdown

[cderv]

The conflicts with latex-divs.lua is now resolved as bookdown lua filter is added before the other and it deals with data-latex attribute if it sees it on one of the supported Divs classes, and removes it so that latex-div.lua in rmarkdown is not activated.

[cderv]

This must have an id thing is really annoying ! But it is currently necessary for the labelling and internationalization for all eng_theorem related chunks.

Ok I now fixed that by generating an id for a div if none is provided, as knitr:::unnamed_chunk() does.

It means we can write a fenced div with no id and it will render correctly in bookdown - parse_fig_labels will do its trick. Obviously, an id should be provided to be able to reference correctly using \@ref

Converting 02-components.Rmd in inst/examples using convert_to_fenced_div() then rendering the bookdown book with version of this branch gives the correct result. The pandoc divs will just use <p> tags for the content, which we did not have before - so some spacing can appear differently.

Using engine:

After conversion, using fenced div:

This seems to impact the inserted blank square. So I need to look at that.

[tchevri]

awesome - and interesting (if not weird) issue... I believe most of us will be keen to know how you fix it, does not look so obvious, why it even happens. Thank you so much for all this effort - we all very much appreciate.

…

On Tue, Sep 22, 2020 at 3:56 AM Christophe Dervieux < ***@***.***> wrote: This must have an id thing is really annoying ! But it is currently necessary for the labelling and internationalization for all eng_theorem related chunks. Ok I now fixed that by generating an id for a div if none is provided, as knitr:::unnamed_chunk() does. It means we can write a fenced div with no id and it will render correctly in bookdown - parse_fig_labels will do its trick. Obviously, an id should be provided to be able to reference correctly using ***@***.*** Converting 02-components.Rmd in inst/examples using convert_to_fenced_div() then rendering the bookdown book with version of this branch gives the correct result. The pandoc divs will just use <p> tags for the content, which we did not have before - so some spacing can appear differently. Using engine: [image: image] <https://user-images.githubusercontent.com/6791940/93814534-ef45c180-fc54-11ea-8164-6eacc49af399.png> After conversion, using fenced div: [image: image] <https://user-images.githubusercontent.com/6791940/93814592-084e7280-fc55-11ea-9f37-bbbd1e5fec7f.png> This seems to impact the inserted blank square. So I need to look at that. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#940 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGOTT66TGXNFBBRX2VH2YYTSG6VXNANCNFSM4QQUIUNA> .

[cderv]

it happens because of the css rule currently used.

bookdown/inst/resources/gitbook/css/plugin-bookdown.css

Lines 93 to 96 in 57f1217

	div.proof:after {
	content: "\25a2";
	float: right;
	}

As using pandoc, everything is inserted between a <p> tag, this rule creates a floating square that is not placed where we want. Modifying the rule to add to the last <p> child should work for the new syntax, but there won't be a square printed in html for the old syntax... Not sure it is bug loss...

[cderv]

Ok this currently fails with pandoc 2.7.3 😞 - I was developing with 2.9.2.1 - I need to find why.
Testing on several pandoc versions like with Rmarkdown would be beneficial...

[cderv]

Ok I tested several pandoc versions and I isolated the issue to be with pandoc 2.7.3 and earlier. It works from 2.8+

From there, following the release note for 2.8, I found that what is described in the documentation about Attributes element (https://pandoc.org/lua-filters.html#type-attr) only work for pandoc 2.8+

That means we need to explicitly use pandoc.Attr constructor and not the simpler syntax {id, classes, attributes}.

It should work now with pandoc 2.7.3 - that was not easy to find as no explicit error, and it is not mention in the documentation.

[cderv]

@tchevri I was looking back at your comment about beamer support, but I can't make the current theorem or proof engines work with beamer output. So I guess I won't deal with this here. could you open a new issue specific to your beamer support idea with a working example so that I better understand what is required. ok? Thank you !

[cderv]

@yihui I think you can review now.

• There is still the square placement for proof that won't work for previous book.
• Where should we document this ? Bookdown book ?
• We don't yet support epub, but I guess you can review what already works.

I tested on several pandoc versions, and with the bookdown book.

Let's get your feedback before going any further - I already spent a lot of time on this one.

About the feature:

• It will work with all the current supported environment, by applying the same style as now
• For TeX, restore_block2 has been adapted. to insert the correct preamble if the filter inserted one \begin{} with one of the supported env
• Referencing is still done by bookdown in R - one contribution above is doing it in Lua and that could be interested to add later
• Internationalization is still done in R for Theorem-like environment and in the Lua filter for Proof-like environment. This is because for the former, it is done by the referencing mechanism in post processing. For the latter, it was done by eng_proof so it was required to change. We could merge the two behaviors eventually and consider them all as the same kind of environment - it is done like that is the contributed filter in one of the comment above
• There is a function to convert a file to use the new syntax from the old syntax
• It work currently for pdf_book, pdf_document2, html_document2, html_book, gitbook

[cderv]

Oh and one think I observed about the current behavior: When rendering to HTML, CSS for those environments like div.theorem are in plugin-bookdown.css used in gitbook() but not in html_document2. That means we don't get any styling for those environment by default. We could easily have a default. I'll open an issue maybe about that.

[atusy]

Good job! I briefly reviewed the lua filter, and suggest changes.

[yihui]

Sorry for the long delay! I just finished reviewing and tweaking this giant PR. I'll merge it for now, and take a look at the CSS issue later. Thanks!

[yihui]

This has been addressed by c59e3dc.

[cderv]

I kind of remember having tried that, and that was not ok in the output... I may have wrongly test it maybe. 🤷‍♂️

[cderv]

Thanks! That was interested to look at the tweak you made to modify this PR.

Even if you did not review the lua file per-se, I guess you've tested the filter itself and that worked ok for you ? Or you blindly trusted me with the general behavior of this new feature ? 😅

take a look at the CSS issue later.

I opened an issue to track this: #977 but it seems you closed it already.

And added another one for beamer support #978

[yihui]

I only tested it briefly (in the bookdown-demo project).

[cderv]

A special option is supported ("bookdown.output.markdown") to output as md... not sure we want to keep doing this.

OK I finally found where blogdown.output.md is used: it is in blogdown
https://github.com/rstudio/blogdown/blob/master/inst/scripts/render_page.R#L10

We may want to check if it works with this new env once we support markdown output in the Lua filter.

[yihui]

This option was introduced in e214250. The main purpose was to support theorem environments for the file format .Rmarkdown in blogdown. This is a rather minor use case. I doubt if anyone has written a single theorem in a .Rmarkdown post :)

That said, it will still be nice if the new syntax works for .Rmarkdown posts. Currently there is a missing step that I haven't done in blogdown yet: .Rmarkdown doesn't support bibliography. That will be implemented by converting .markdown to a certain Markdown output format. Theorems in fenced Divs could be converted to HTML in this step, too (::: needs to be converted to <div>).