quarto render in parallel across multiple cores?

[paulgribble]

Is it possible (or can it be made possible) to execute quarto render in a project directory with many many files so that it makes use of multiple cores? (e.g. in the same way one can use the -j flag with the make command to simultaneously run independent targets?) thanks,

[cscheid]

Not right now, but it's in our medium-term plans (I would guess later than next month, but not later than the next decade).

[eitsupi]

Parallelization with make may fail in the process because intermediate files may be created with the same name.
See #731

[cscheid]

Yes, we are aware of that issue. To add a bit more information for why this is a real challenge:

1. we need to refactor our TypeScript codebase to not block on rendering each part (laborious, but not particularly hard)
2. we need to carefully catalog and track filesystem dependencies and side-effects of our toolchain, at least Jupyter, Knitr, Pandoc, LaTeX (laborious and hard)
3. some of our generated files have implied dependencies (eg listings pages need the items to be rendered ahead of time). This will probably mean writing a custom scheduler (laborious, but not particularly hard)

[venpopov]

any updates on this?

[jamesowers-cohere]

Is there a place where we can track issues related to this? Currently having to roll my own wrapper for quarto to handle dependency specification, and parallelisation, so it would be great to see what work is being done on this such that I can make a contribution to your project.

[cscheid]

#6518

[ppm1337]

You can achieve this by copying everything you need (*.qmd, *.yml, *.css, etc.) to a separate location (e.g. a temp folder) for each set of parameters and render your report/project there.

I frequently generate analysis reports in either R or Python for many samples in parallel using this approach (the code is more involved but I tried to make it as minimal as possible, here only for a single *.qmd file):

In R:

library(future)
library(purrr)
library(furrr)

# create basic parameterized quarto script
qmd_path <- "hello.qmd"
qmd_code <- "
---
title: Greeting
format:
    html:
        embed-resources: true
params:
    name: ''
engine: knitr
---

Hello, `r params$name`!
"
invisible(file.create(qmd_path))
writeLines(qmd_code, qmd_path)

# define render function
render_qmd <- function(
    qmd_path,
    id,
    params,
    render_dir = tempdir(),
    publish_dir = "~/results/notebooks"
) {
    stopifnot(endsWith(qmd_path, ".qmd"))
    stopifnot(file.exists(qmd_path))

    # create unique temp dir for the given ID
    id_dir <- file.path(render_dir, id)
    if (dir.exists(id_dir)) {
        unlink(id_dir, recursive = TRUE)
    }
    invisible(dir.create(id_dir))
    # optional: give qmd file a unique name (and with it the resulting HTML file)
    id_qmd_path <- file.path(id_dir, sub(".qmd$", sprintf("_%s.qmd", id), basename(qmd_path)))

    # copy the qmd file to the unique (temp) dir
    file.copy(qmd_path, id_qmd_path, overwrite = TRUE)

    # copy other required files (e.g. your custom css)
    # assuming you store them in the same dir as the qmd file
    css_files <- qmd_path |>
        dirname() |>
        list.files(pattern = "\\.s?css$", full.names = TRUE)

    if (length(css_files) > 0) {
        walk(css_files, \(f) file.copy(f, id_dir, overwrite = TRUE))
    }

    cmd_params <- params |>
        imap_chr(\(v, n) paste0("-P ", n, ":", v)) |>
        reduce(paste)

    # construct quarto CLI command
    cmd <- sprintf(
        "quarto render %s %s --execute --output-dir %s",
        id_qmd_path,
        cmd_params,
        publish_dir
    )
    # debug
    # print(cmd)
    # list.files(id_dir)

    # if one fails, don't interrupt the whole process
    safely(system, quiet = FALSE)(cmd)

    # clean up
    unlink(id_dir, recursive = TRUE)
}

# render in parallel
names <- c("Alice", "Bob", "Charlie")
plan(multisession, workers = I(length(names)))

future_walk(names, \(n) {
    render_qmd(
        qmd_path = qmd_path,
        id = n,
        params = list(name = n)
    )
})