Questions about freezing behavior

[andrewheiss]

(This is all with Quarto 1.3.49 on macOS Ventura)

This might be me just misunderstanding what freezing does, but I'm running into slower site building times even with freeze enabled because Quarto works through all 100+ qmd files in one of my projects when I run quarto render at the terminal, even though nothing has changed in the source code.

It might also be that I'm misinterpreting what Quarto means by "render" when compared to other systems like blogdown. Blogdown has an argument in its build_site() function that filters the list of Rmd files to re-knit. build_site(build_rmd = 'md5sum'), for instance, will keep track of the MD5 checksums of all the Rmd files and only re-knit those that have changed. In their documentation, they call this "rendering"—any files that aren't modified don't even get looked at when the site is building.

In the documentation for Quarto's freeze, it also talks about selectively re-rendering documents, but in practice, rendering seems to refer to running computational chunks, not converting markdown to HTML/PDF/whatever. As a result, from what I see when I build projects, Quarto still loops through every qmd file and reconverts them to HTML/PDF/etc., but skips the computational output stored in _freeze

Here's a reproducible illustration of what I'm talking about to clarify things. It's based on this zipped up mini website: blog-example.zip. freeze: true is enabled in _quarto.yml.

Initial project building before _freeze exists

Running quarto render for the first time loops through all 5 .qmd files in the project and runs their chunks through R. Quarto stores the output and some corresponding JSON in the _freeze folder at the root of the project, as expected:

Terminal output from quarto render, showing 5 documents getting rendered

% quarto render 
[1/5] posts/third-post/index.qmd


processing file: index.qmd
  |.......................                                               |  33%
  ordinary text without R code

  |...............................................                       |  67%
label: unnamed-chunk-1
  |......................................................................| 100%
  ordinary text without R code


output file: index.knit.md

[2/5] posts/first-post/index.qmd


processing file: index.qmd
  |.......................                                               |  33%
  ordinary text without R code

  |...............................................                       |  67%
label: unnamed-chunk-1
  |......................................................................| 100%
  ordinary text without R code


output file: index.knit.md

[3/5] posts/second-post/index.qmd


processing file: index.qmd
  |.......................                                               |  33%
  ordinary text without R code

  |...............................................                       |  67%
label: unnamed-chunk-1
  |......................................................................| 100%
  ordinary text without R code


output file: index.knit.md

[4/5] index.qmd
[5/5] about.qmd

Output created: _site/index.html

That's great and what I expected.

Project building after modifying one frozen file

If I modify posts/first-post/index.qmd (add a space or newline or text or something somewhere in it) and then run quarto render, I would expect Quarto to only re-render that file. Instead, Quarto works through all five qmd files. The computations from the unmodified files aren't run again and Quarto only re-knits the single modified file, but still goes through the effort of checking all the other files and converting them from (q)md to HTML:

Terminal output from quarto render, showing 1 document getting re-knit and the other 4 getting re-built

% quarto render
[1/5] posts/third-post/index.qmd
[2/5] posts/first-post/index.qmd


processing file: index.qmd
  |.......................                                               |  33%
  ordinary text without R code

  |...............................................                       |  67%
label: unnamed-chunk-1
  |......................................................................| 100%
  ordinary text without R code


output file: index.knit.md

[3/5] posts/second-post/index.qmd
[4/5] index.qmd
[5/5] about.qmd

Output created: _site/index.html

If I run quarto render again immediately after, with no file modifications, Quarto still works through all 5 documents:

Terminal output from quarto render, showing no documents getting re-knit, but all 5 are re-converted

% quarto render
[1/5] posts/third-post/index.qmd
[2/5] posts/first-post/index.qmd
[3/5] posts/second-post/index.qmd
[4/5] index.qmd
[5/5] about.qmd

Output created: _site/index.html

This is where I'm likely conflating Quarto's freezing with blogdown's MD5 filtering. In blogdown, only the one modified file would be both reknitted and converted to HTML—all the other files would be ignored and not even looped through.

And this is where the site-building slowdown is happening for me. In a project I have with 100+ qmd files, Quarto works through each individually. Chunk computation doesn't happen again if the files are unmodified, but it takes a couple minutes to check and rebuild each of the files, even though nothing has changed.

But maybe this is what freezing is designed to do? It keeps track of changes so that computational chunks don't have to be rerun, but the overall pandoc conversion to HTML still seems to happen every time?

Project previewing after modifying one frozen file

What also might be confusing me about what freezing is supposed to do is what happens when using quarto preview. With the ongoing reproducible mini site example, run quarto preview in the terminal and make a small change in posts/second-post/index.qmd. Quarto will automatically rerender the site, but this time it will only work through a list of two qmd files: the modified blog post file and the listings page. All other files are untouched.

Terminal output from quarto preview, showing two documents getting re-rendered after one is modified

% quarto preview
Preparing to preview

Watching files for changes
Browse at http://localhost:7318/
GET: /
[1/2] posts/second-post/index.qmd


processing file: index.qmd
  |.......................                                               |  33%
  ordinary text without R code

  |...............................................                       |  67%
label: unnamed-chunk-1
  |......................................................................| 100%
  ordinary text without R code


output file: index.knit.md

[2/2] index.qmd

GET: /posts/second-post/

This behavior is what I think I've been expecting freezing to do—that Quarto would only work through a list of modified files (in this case 2) instead of a list of all the files (in this case 5)

---

General overarching question

Is there a way to get Quarto to only re-convert (or loop through) modified files, similar to what happens with quarto preview? That is, is there some way to make it so that if I modify one file, like a blog post, quarto render will only deal with 2 files instead of all 5? Like, if I modify posts/second-post/index.qmd and run quarto render, the output would look like this:?

% quarto render
[1/2] posts/second-post/index.qmd
[2/2] index.qmd

[bryanwhiting]

No answer, but I had the same idea: #3674

[mcanouil]

I think we have a misconception about what freeze is about.
Regardless of the presence of code cells, all files needed for the website are rendered from what I've seen.

Which means for example, within a "prose" only website, with new content comes extra rendering time.
execute.cache won't help either.
Maybe execute.enabled: false can shorten rendering to HTML time.

If I am not mistaken, this means there is a "missing" feature allowing the user to not render at all to HTML if there is no reason, i.e., not only the content of the "page" but any Quarto component involved in that page such as navbar, footer, etc.

[andrewheiss]

Yeah, I think there's a mismatch between what blogdown and knitr consider "rendering" and what Quarto considers "rendering". With blogdown, unmodified files don't go through pandoc when rebuilding the site; with Quarto, unmodified files (even when their computational output has been frozen) still go through pandoc

[mcanouil]

Indeed, but with execute.freeze: true the code cells are not executed at least, i.e. qmd -> md.
Only md -> html part is every time.

[cscheid]

@andrewheiss Yes, that's true and we should go through our docs to make sure that we're being clear about the distinction. For what's worth, the reason this happens is that quarto has more "dynamic" features which can cause outputs to change even if files don't (eg. listings pages). Working out dependencies more carefully is on our radar, but there's a lot of other work we need to do ahead of it, so it's not going to be something we'll be able to fix in the short term.

[mcanouil]

I am going to quote @cscheid (#3674 (comment)):

With regards to re-rendering, we're currently being conservative on quarto render because of possible dependencies. As a simple example, listings can use content from other files, and so a simple diff-based approach won't always work.

If you're working interactively on a large quarto project, then quarto preview can offer a better experience than quarto render everything all the time: the preview command will render everything once, but then will only refresh the changed files.

[jarbet]

I'm confused as how to address the original question. I put the following in my project _quarto.yml file:

execute:
  freeze: auto

However, when I do quarto::quarto_render() in R or quarto render in the terminal, it reruns all of my .qmd files which is time-consuming. Same if I use freeze: true.

Maybe I'm misunderstanding the purpose of the freeze feature, but the documentation strongly suggests that it can be used to control which .qmd files are rerun when rendering - but the feature doesnt seem to work since all .qmd files are always rerun?

[mcanouil]

Please open a new discussion with a fully reproducible small example depicting what your project looks like.

This thread is almost 3-years old.

[jarbet]

Please open a new discussion with a fully reproducible small example depicting what your project looks like.

This thread is almost 3-years old.

I updated to the latest version of Quarto and I think the feature is mostly working now. Please ignore my above comment.