Having Quarto (or Pandoc) use more cores / threads ?

[ma-riviere]

Hello Quarto,

I'm wondering if there is a way to make Quarto (or Pandoc) use more threads/cores ?

I'm currently compiling a document to HTML (from R), and it takes more than 3 hours. The document is not that long: 2700 lines (more than half of those being text). There are no complex computations (the knitting in itself takes only a few minutes), but once it's done knitting, it will take close to 3 hours to output the HTML document (which is roughly 15MB in size).

During that whole time, only ~5% of my CPU is used by the Deno virtual machine, so I was wondering if there was a way to have it use more CPU resources to speed up the process ?

Thanks :)

[jjallaire]

Using more cores isn't the problem here and none of the operations that transform .md to .html are fundamentally parallelizable. It's possible depending on what's in your document that its waiting on network IO? I think to really help here we need to get a reproducible example then we can dig into exactly what's happening.

[ma-riviere]

@jjallaire Thanks for your input !

The document is a "tutorial" on data wrangling. It lists equivalences between dplyr/tidyr and data.table for most data wrangling operations. There shouldn't be any IO in there, and all the datasets used in the examples are created in the doc itself.

You can see it on RPubs. There are a few hidden code chunks at the beginning, that define a custom knit_print for data.frame to have collapsible & auto-truncated gt tables as output for data.frames, but aside from that, everything else in the document is visible.

Edit: currently trying to compile this document with Quarto 0.9.449 on Ubuntu 20.04 (WSL2), and the last time I compiled it was on Windows 11 (don't remember the version of quarto, should have been around 0.9.350).

[cderv]

since the knitting in itself only takes a few minutes (what I call knitting is the phase before it displays "output file: datatable.knit.md"), and I thought that those custom printers were applied during that phase, not after (after that point, I thought it was the md -> HTML conversion by Pandoc).

Regarding your thoughts, you're right to think that knitting will happens and then other process not involving the knitting. So yes the custom printer makes the knitting slower (probably because of all the cat() you are doing, or something else) but if you get a very slow rendering after the knitting has happened, then it would mean there is indeed something else.

Step you could do to simulate what happens is either knitr::knit() your file to see if the custom printer really takes a lot of time, or if you did use many Quarto feature in chunks call rmarkdown::render() to use R Markdown instead of Quarto - both of them would give some hint if Quarto layer is adding much or not to the timing.

Having your document in a github repo, possibly with the renv.lock file, would help us reproduce (probably by minimizing the size also but I can do that). For now I got a "function measure() not found" 🤷‍♂️ so probably something I am missing.

[ma-riviere]

@cderv I'm going to try to knitr::knit() & rmarkdown::render().

"function measure() not found"

You need data.table 1.14.3 to use that function (and IIRC, the CRAN version is still 1.14.2) -> install "Rdatatable/data.table"

Maybe also remove the .libPaths mess in the first chunk: Quarto doesn't find my libraries when I render without that, but that's probably a me-specific issue (one that I will open another day).

[ma-riviere]

• knitr: completed in a few minutes, gives me a 107 KB .md file. Also, while knitting, it printed every single table as a shiny.tag element underneath the .Rmd cell I executed the knitr::knit() from.

• rmarkdown: also finished rendering in a few minutes, but it doesn't display my tables at all. Instead, I get something like this:

## 
## <details>
## <summary>
## *data.table [32 x 11]*</summary>
## <br>
## <div class='light-mode'></div><div class='dark-mode'></div></details>

So, I might need to modify either the YAML options, or my custom printer to be able to get the same result as Quarto ? The output HTML is only 2.3 MB due to that.

[cderv]

This is probably because you did not adjust the Quarto specific option like

execute:
  warning: false
  message: false
  output: asis
  freeze: true

This is not supported by R Markdown, and you'll need to call knitr::opts_chunk$set() in setup chunk to change default chunk options for all your chunks.

knitr::opts_chunk$set(warning = FALSE, message = FALSE, results = "asis")

You could also have used knitr::asis_output() in your printing function (https://cran.r-project.org/web/packages/knitr/vignettes/knit_print.html#for-package-authors)

Anyway, I suggested that to understand timing better and as a way to know how quarto was involved exactly in the timing.

[ma-riviere]

Thanks ! I've been using Quarto so much lately that I forgot my rmarkdown basics ...

So, I made a .Rmd out of the .qmd and adjusted the YAML to get a proper HTML output and ... it works. Rendered in a few minutes.
It's uglier than Quarto, but the tables are there, collapsible + light/dark mode & all (well, except that both dark & light tables are displayed simultaneously, but that's to be expected).
HTML file is 16 MB (the version of the file that I'm rendering has additional sections compared to the one that's online, hence the 1 MB difference IMO).

PS: should I (or one of you ?) make a new GH issue out of this since it seems it is a Quarto issue in the end ? I initially thought this rendering time was normal (considering the number of tables involved), and I only opened this "Q&A" to see if I could make Quarto/Pandoc use more of my CPU to hasten things up. But we kind of went on a big tangent here (compared to the thread's title) ^^'

[cderv]

What happens if you don't use self-contained document which is the default I think ? Setting self-contained: false ?

Making a self contained HTML can take time as everything will be encoded in the single HTML file. It is also quite heavy (15MB in size for a HTML file is heavy.) Without self-contained, the file will be lighter, and it could be quicker, but you'll have external resources probably and/or you'll need online access.

Can you try that ?

As there is a lot outputs (like gt tables), this could also come from all our processing after the computation step to support different features of Quarto. 🤔

[ma-riviere]

Thanks for your input !

Currently trying, it has already been running for 35 minutes (it slowed down at the same step as usual).

For now, it doesn't seem any of the tables are saved as external files (I only see the 6 very small plots that I have within the page being saved in the *_files folder). For reference, I have another quarto document that ends up as a > 10MB HTML (mostly due to the plots it includes, it only has a few tables), and that document is compiled in a few minutes. So it's probably due to the (many) tables (like you mentioned), not the size of the output.

PS: due to me using custom knit_print methods (with HTML tags directly included), all the chunks of the "problematic" document have output: asis (if that can help explain the issue).

[ma-riviere]

Update: it finished rendering in ~ 1h10 with self-contained: false, but (I forgot) I had commented the part that duplicates every table (to have one for both light and dark modes), which probably explains most of the speedup (and also why the resulting file is half the usual size).

[jjallaire]

Our Lua filters do apply patterns to HTML tables so that they can apply labels and captions from YAML. What happens if you remove any labels or captions you have?

@cscheid I don't think this can just be O(n) in the print method if knitr is completing fairly early in the render.

[cscheid]

I agree. I think there's more than a single issue going on.

In my testing I hadn't let knitr run to completion with the custom printers because it took longer than a minute, and I could see each subsequent cell taking longer than the previous. I also had noticed that this didn't happen without custom printers. As a result, I missed that knitr took minutes and something else took hours.

[ma-riviere]

I don't actually have any label of caption in this doc (unless I misunderstood what you meant).

The only YAML headers for chunks that I use are:

• warning: false, message: false, and output: asis applied to every cell (under execute)
• A few echo: false and output: false here and there (in specific chunks), and a few results: markup to override the output: asis in the few cells I don't want asis output in (I didn't find a better way to do this).

[cscheid]

Here's some more data as I run it.

I created a "manageably bad" version of this file by rendering only the first 500 lines or so. I get the following overall timing:

self-contained?	custom printers?	overall clock time
false	false	13.28s
true	false	15.6s
false	true	Still running..
true	true	Still running..

That version with self-contained: false and with custom printers is still running after a good amount of time. I'm going to abort the experiment and try an even smaller version.

Even smaller version

self-contained?	custom printers?	overall clock time
false	false	6.82s
true	false	7.67s
false	true	48.12s
true	true	47.42s

So it's not self-contained: true

[cderv]

The custom printers will definitely come in play in the process because I believe they are the one formatting the data.frame as gt HTML table. Without that, I bet the resulting HTML would be way simpler.

It would be interesting to have some debug output in our code to get the steps where the rendering is at so that we can indeed know if the post knitting step is taking a long time or not. In R Markdown we have some calls to perf time function activated in each step to get a summary of time for each (pre-knit, knit, post knitr, pre pandoc, pandoc convert, post pandoc).

Let's wait for the two other results that will tell us if Pandoc is taking a long time to make the document self contained.

We would need one more comparaison in this table to see if our filters step are the ones taking the times processing all the HTML code of the document.

[ma-riviere]

@cderv Yes, for each data.frame object, the custom printers do:

• Transform it into two gt() object, and apply both my light and dark mode styles to them (each get put in their own <div> with the related CSS class, and my custom .scss themes mask one or the other when I trigger the dark mode button of Quarto).
• Wrap both within a <details> tag to have the collapsible result (that mimics the description of a data.frame when displayed inline in RMarkdown)

[cscheid]

I'm on a call with Rich Iannone, the author of GT, and he suggested replacing gt::tab_style with gt::tab_options, to avoid repeating the inline styles: https://gt.rstudio.com/reference/tab_options.html

There's definitely more to it and I'll have more to say in a couple of hours when my granular timing runs are done. Stay tuned!

[cscheid]

(cc @rich-iannone)

[ma-riviere]

@cscheid
@cderv
@jjallaire

Tagging everyone here to let you know about the most recent development: using rmarkdown to render an .Rmd equivalent of my .qmd document works, and renders properly in a few minutes (see this comment). So it would seem that this is Quarto-related in the end ?

As I mentioned in the linked comment, should we move this to a specific GH issue (since I didn't think it was an issue when I opened this Q&A) ?

Also, thank you all for your reactivity & helpful suggestions. This is the best 'getting help to solve an issue on an open-source software' experience I've ever had !

[cderv]

@ma-riviere there is now an issue to follow this as an issue #1152 as we need to look into that deeper. (Issue created from thread is shown on the right column at the bottom)

[ma-riviere]

I had missed that, thanks.
Good luck for solving that issue, and thanks for all your work !

[cscheid]

@ma-riviere Take a look at some of the workarounds I found here: #1152 (comment). We're fixing some of the problems in quarto and upstream libraries, but a full treatment will unfortunately take some time.

[ma-riviere]

I'm going to render the whole document with self-contained: false & no code linking, and see how long it takes (compared to the usual ~3 hours).

Going to mark your reply as answer since I think it's the closest I'll get to a working fix in the immediate future.

Thank you (all of you) for your help :)

[kwstat]

Here's a self-contained example: https://github.com/kwstat/quarto_test
There are (essentially) identical .qmd and .Rmd files. I am using RStudio 2022.07.1+554 with Windows 10. The timings are done with a simple stopwatch on my phone.

The .Rmd file renders to pdf in 0:51.

The .qmd file renders to pdf in 3:10.

[cscheid]

I'm running our profiler to see if there are easy performance wins here, but 1MB documents are not exactly a common use case for quarto! ;)

[cscheid]

On my local machine (a 2-year old Intel MacBook Pro), this is what I get:

format	time
HTML	15.4s
PDF	31.2s

[cscheid]

Our windows expert is out on vacation, but I'll follow up with him when he's back.

[cscheid]

@kwstat It's possible that you're running a version of quarto that definitely had a performance issues on larger files. If you don't see something like

WARNING: Warning: diff of engine output timed out. No source lines will be available.

on the output generated by quarto, then you're running a version that is affected by this issue that we recently fixed.

[jjallaire]

Also note that Quarto xelatex by default and R Markdown uses pdflatex by default. xelatex is known to be slower, but supports UTF-8 characters and use of local system fonts. You can override this default with:

pdf-engine: pdflatex

[kwstat]

I updated to the latest daily RStudio 2022.07.2-559.
In the .qmd file I changed pdf-engine: pdflatex
Crude timings from stopwatch.

.Rmd to .pdf:

0:00 Start
0:06 md created
0:50 pdf created

.qmd to .pdf

0:00 Start
0:08 md created
0:45 pandoc starts
2:02 pdflatex - 1
2:24 pdflatex - 2
2:44 pdf created

It doesn't look like xelatex or pdflatex is to blame. Maybe the new pandoc filters are slow???

[cscheid]

What's the output of quarto version for you? I think this fix is only in a prerelease version currently.

[kwstat]

quarto --version
1.0.38

[cscheid]

Ok: we fixed a significant performance issue around 1.1.35. If you can install the latest prerelease version of quarto, you will probably see performance numbers closer to mine.

[kwstat]

OK, I installed RStudio Daily 2022.11.0-daily+101

qmd to pdf

0:00 Start
0:13 md done
0:56 "metadata" printed
2:03 pdflatex -1
2:24 pdflatex - 2
2:44 pdf done

Again, md creation is fast and pdf creation time is okay, but in-between is slow, perhaps pandoc is the problem?

[jjallaire]

Installing the RStudio daily isn't enough, you should update Quarto the pre-release: https://quarto.org/docs/download/prerelease.html

[kwstat]

Just an update on the timings. Also, deno.exe seems to take a long time. Maybe my corporate-managed laptop has security software that is scanning deno.exe when it runs (or some such thing).

Using this test case (creates 480 page pdf): https://github.com/kwstat/quarto_test

With this configuration : Windows 10. RStudio 2022.12.0 Build 353, R 4.2.2

I am still seeing slow rendering times for qmd files:

qmd -> pdf   2:40  (2 min, 40 sec)
Rmd -> pdf   0:53  (0 min, 53 sec)

qmd header:
format: pdf
pdf-engine: pdflatex

Rmd header:
output: pdf_document

qmd warning
WARNING: Warning: diff of engine output timed out. No source lines will be available.
pandoc 
  to: latex
  output-file: quarto_test.tex
  standalone: true
  pdf-engine: pdflatex
  variables:
    graphics: true
    tables: true
  default-image-extension: pdf

$ quarto --version
1.2.269

[cscheid]

WARNING: Warning: diff of engine output timed out. No source lines will be available.

This is the culprit right here. One of the main differences between quarto and rmarkdown is that we put a lot of engineering effort explicitly in generating good error messages, which includes good line number information. Unfortunately, knitr doesn't currently offer this information, which means we attempt to recover it via a diffing operation. In larger files, this diffing operation takes too long, and so we have a timeout.

The right fix here would be to speculatively execute the code assuming it won't fail, and then lazily attempt to compute the diff if an error happens. But that is a big architectural change for our code, so it'll have to wait.

[jimgar]

@cscheid I'm also getting the diff of engine output timed out warning with R code.

Speed isn't that important to me. The report gets rendered probably within 15 seconds and I'm fine with that.

I wish I could share my work as an example but it uses an internal company R package for interacting with our database, and the data from said interaction forms the basis of all of the downstream work. Plus the data itself is sensitive for our clients.

Does the warning mean knitr is throwing an error at some point during rendering? Or could it also be a warning? An error would be weird because I still get a rendered HTML doc at the end. It contains all the content I'm expecting to see. The only thing is that a dataset download button no longer works when clicked. This only seem to happen for larger clients, but I can't reliably predict what the cut-off size seems to be. Maybe that's it?

Without any informative error messages I'm at a bit of a loss with what to try in terms of debugging further. I do notice that the gt package seems to be a culprit in other people's experiences - and I've got quite a few tables being made with that.

Very sad way to end the week :(

[cscheid]

Does the warning mean knitr is throwing an error at some point during rendering?

No. The error message says: "diff of engine output timed out. No source lines will be available." It means that the numbering of source lines is not going to be tracked precisely. Your output is unaffected.

The only thing is that a dataset download button no longer works when clicked.

That's almost certainly a different problem, unrelated to this issue. Unfortunately, if you can't share the source, we can't diagnose the problem...

[jimgar]

Ok, pretty much what I thought, and I understand. Trust me, I wish I could just share. Thanks for responding.