Race condition when running multiple instances of quarto

[stefanv]

Bug description

When running several instances of quarto render in parallel, errors like these appear:

ERROR: NotFound: No such file or directory (os error 2): copy '/home/stefan/.local/quarto-1.4.76/share/formats/html/quarto.js' -> '/home/stefan/p/resampling/resampling-with/source/site_libs/quarto-html/quarto.js'

NotFound: No such file or directory (os error 2): copy '/home/stefan/.local/quarto-1.4.76/share/formats/html/quarto.js' -> '/home/stefan/p/resampling/resampling-with/source/site_libs/quarto-html/quarto.js'
    at Object.copyFileSync (ext:deno_fs/30_fs.js:80:7)
    at copyFileSync1 (file:///home/stefan/.local/quarto-1.4.76/bin/quarto.js:36992:10)
    at copyTo (file:///home/stefan/.local/quarto-1.4.76/bin/quarto.js:36937:9)
    at doCopy (file:///home/stefan/.local/quarto-1.4.76/bin/quarto.js:36967:9)
    at copyFileIfNewer (file:///home/stefan/.local/quarto-1.4.76/bin/quarto.js:36984:9)
    at copyDependencyFile (file:///home/stefan/.local/quarto-1.4.76/bin/quarto.js:57728:5)
    at copyFile (file:///home/stefan/.local/quarto-1.4.76/bin/quarto.js:57666:13)
    at file:///home/stefan/.local/quarto-1.4.76/bin/quarto.js:57673:50
    at Array.forEach (<anonymous>)
    at processHtmlDependencies (file:///home/stefan/.local/quarto-1.4.76/bin/quarto.js:57673:32)

Steps to reproduce

I don't have a very good way to reproduce, unfortunately, other than to build several chapters of a document in parallel. I hope the problem is straightforward to identify :/

Expected behavior

No response

Actual behavior

No response

Your environment

No response

Quarto check output

$ quarto check

[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.1.2: OK
      Dart Sass version 1.55.0: OK
      Deno version 1.33.2: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 1.4.76
      Path: /home/stefan/.local/quarto-1.4.76/bin

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.11.1
      Path: /home/stefan/envs/py311/bin/python3
      Jupyter: 5.3.0
      Kernels: python3

(-) Checking Jupyter engine render....0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[✓] Checking Jupyter engine render....OK

[✓] Checking R installation...........OK
      Version: 4.1.3
      Path: /usr/lib64/R
      LibPaths:
        - /home/stefan/R
        - /home/stefan/R-dev
        - /usr/lib64/R/library
        - /usr/share/R/library
      knitr: 1.43
      rmarkdown: 2.22

[✓] Checking Knitr engine render......OK

[mcanouil]

Without explaining clearly the steps to reproduce and what you mean by parallel, I'm afraid it isn't "straightforward to identify".

If I had to guess, you are rendering the same thing multiple times and concurrently. If that's the case, you can't do that because there are temporary files "attached" which get deleted at the end of rendering thus, could be deleted while other files still rely on them.
Again, without a clear depiction of your setup/use case, we are unlikely to provide further assistance.

[stefanv]

I think you already touched on part of the problem: possible dependence between documents triggering overlapping builds.

What I'd like to do is parallelize the building of chapters of a book.
The way I am doing it right now is to use quarto render chapter.Rmd on each (all in parallel).
From what you write above, I suspect that quarto render chapter.Rmd may build more than one chapter at a time, perhaps if chapters refer to one another?

So perhaps the question I should be asking is whether there currently is a reasonable way to parallelize builds.

[cscheid]

@stefanv There isn't one. This is a known limitation in Quarto, and one we're going to address in 1.5. If you absolutely need parallel rendering, the current best practice is to do so manually in a pre-render script in a manner that's completely managed by yourself, and that doesn't involve parallelizing the calls to pandoc.

[mcanouil]

FYI:

• Parallel Computing - using one .qmd to render n output files - cannot open the connection #5861
• Parallel creation of parameterized reports fails due to file conflicts #4730