Long render times

[jjallaire]

Discussed in #1149

^{Originally posted by ma-riviere June 14, 2022}
Hello Quarto,

I'm wondering if there is a way to make Quarto (or Pandoc) use more threads/cores ?

I'm currently compiling a document to HTML (from R), and it takes more than 3 hours. The document is not that long: 2700 lines (more than half of those being text). There are no complex computations (the knitting in itself takes only a few minutes), but once it's done knitting, it will take close to 3 hours to output the HTML document (which is roughly 15MB in size).

During that whole time, only ~5% of my CPU is used by the Deno virtual machine, so I was wondering if there was a way to have it use more CPU resources to speed up the process ?

Thanks :)

[cscheid]

Collecting data

Here's a partial flamegraph of a smaller version of the file @ma-riviere shared with us:

One of the culprits is a call to https://github.com/kpdecker/jsdiff, which is taking 20s to diff a 500LOC file against a 17kLOC file with 260 total changes. That's surprisingly large. In contrast, a call to unix diff with the same inputs is instant.

[jjallaire]

You could just set a threshold for the file sizes for which we'll try to do this mapping?

[cscheid]

Updates:

• We found a number of performance issues in GT which we need to fix, but those will unfortunately take a while.
• We now have a 2s timeout for the diff operation, after which we give up doing source line recovery (post 1.0 I'm confident we can bring it back with a better algorithm).
• code-link is the culprit for the engine-postprocess timing, which is about 10s here. I recommend disabling that for now, because fixing it will be a relatively large task
• Then, removing self-contained also reduces the runtime by ~10s.

Given all these changes, the total runtime is dominated by render-execute, which is knitr:

We also verified through profiling that GT is responsible for the vast majority of the runtime in render-execute.

It's still worrisome that pandoc takes a couple of seconds on this input (that's probably our filters, the profiling of which I haven't enabled yet.)

So, unfortunately, the appropriate resolution here needs to fix a number of different issues.

[cderv]

code-link is the culprit for the engine-postprocess timing, which is about 10s here. I recommend disabling that for now, because fixing it will be a relatively large task

@cscheid about this, is this the part where downlit package is used and some performance issues are with R functions in xml2 and downlit 📦 or other process internal to Quarto ? Just curious as I already notice some improvement to make (not on performance though). Thanks

[grasshoppermouse]

Not sure this belongs here, but in RStudio on macOS, after the qmd file is done rendering, the GUI locks up -- I can't edit text or use the console for 20-30 seconds, even after the rendered html file appears in the Viewer pane.

[cscheid]

@grasshoppermouse If it's an RStudio IDE issue, please file that on the RStudio IDE repository.

[grasshoppermouse]

How can I tell? I've never had this problem with the original knitr/rmarkdown workflow.

[cscheid]

You mentioned the problem is that RStudio on macOS freezes for you after building. That means it's an RStudio issue.

[ma-riviere]

@cscheid It seems the problem is over 😄. Just rendered my "long" document full of tables again, and it barely took a few minutes, with no slowdown after all chunks were parsed. Thanks !

• R: 4.2.1 (2022-06-23)
• OS: Ubuntu 20.04.4 LTS
• Pandoc: 2.19.2
• Quarto: 1.1.251

---

Edit: there's a 'Warning: diff of engine output timed out. No source lines will be available.' that shows when that page is done rendering. Might be linked to this sudden speed gain.

[sophiestrazie]

My problem is related to the run time, but not exactly the same.

My error looks like this:

I am publishing 77 html quarto files to Netlify with each page having at least one image in the ![]() format. To get a sense of the amount of content and whether this is the problem, my website I was able to publish is here. There does not seem to be a solution for non-code related run times. I cannot tell which file is in bin/quarto.js:109513:17 to address it.