[Feature request] Can Quarto please create_all_ temporary files _not_ in the same directory with my source files?

[ForceBru]

What?

I'd like Quarto to create all temporary/intermediate files in the given subdirectory of the project root (or /tmp, or another user-provided directory). By "intermediate files" I mean all files that Quarto manages itself: .aux, .tex, .log, the intermediate .pdf, the .md generated from the .qmd and so on.

I'd like to write something like this in the frontmatter:

project:
  output-temp-dir: temporary_files
  output-dir: out

This would make Quarto write stuff only to output-dir and output-temp-dir and nowhere else. Currently, like knitr/bookdown/Rmarkdown, Quarto will write temporary files to the current directory.

Why?

1. I created a Quarto project with the default file report.qmd (Quarto markdown).
2. I tried editing it in RStudio but didn't like it, so I exited RStudio without closing the file.
3. I switched to Zettlr, but it couldn't see the file, so I renamed it to report.md (regular Markdown). Now it worked and I continued working on report.md for a week in Zettlr.
4. Rendering the file with quarto render --to pdf worked great. I simply had to put project: render: report.md in _quarto.yml.
5. Today I started RStudio and apparently saved report.qmd (Quarto markdown!), so that it appeared in my project. Now I had both report.md with a week's worth of work and the almost empty report.qmd. For whatever reason (again, probably RStudio re-saved the old version of _quarto.yml too) the line project: render: report.md got changed to .qmd.
6. I finally finished my work on report.md and rendered it: quarto render --to pdf.
7. report.md was GONE and report.pdf contained whatever nonsense report.qmd had.
8. I attempted to recover report.md with various tools, but to no avail.

It looks like in the process of analyzing report.qmd, Quarto deleted the "old" report.md, created a new report.md with the processed contents of report.qmd, merrily created the PDF from that MD file and then deleted report.md as an artefact of processing report.qmd.

Now all my work is gone, and it's kinda my fault because I named my file report.md and had a report.qmd alongside it. Also, I could've easily used version control, but deliberately chose not to since I was too lazy to set it up. Which turned out to be a stupid decision. It's also Quarto's fault: it could've looked at the modification dates, wondered why the "temporary" file report.md is much newer than the "original" report.qmd and asked me for permission to overwrite report.md.

---

I would like Quarto (and knitr/bookdown/RMarkdown/whatever) to do its processing and creating temporary files in /tmp, because that's where temporary files are supposed to go. Or in output-temp-dir: temporary_files, as suggested earlier, so that the user can control where the temporary files are going. But not in my working directory, where it can overwrite any important .md file I spent a week working on.

That would solve all such problems: I write my MD files in my project's directory, Quarto copies everything to /tmp/whatevergibberish, processes it and puts the output into output-dir. If the project's settings (or the frontmatter) require to save the .tex file, then copy it into the output directory as well. This way, the user doesn't have to worry about the processing done by Quarto. They also don't need to worry about Quarto overwriting their .md files.

Also, all output only goes to output-dir and Quarto never ever writes to the directory with the user's source files or any other directory besides those marked as output-: output-temp-dir and output-dir.

[cscheid]

I'm really sorry you lost work. That sucks, no two ways about it.

Unfortunately, the solution you're proposing won't work in general, because of a combination of factors:

1. we depend on other software that writes to the current directory (notably, the LaTeX suite)
2. we're executing arbitrary executable content, by design, and we can't know ahead of time which files are generated that are meant to be moved to the output location

We should absolutely have a system to warn about which files are safe to create inside a quarto folder, at least for .md files.

[JonasLoos]

Maybe quarto could do the processing in a separate temp dir, where all the source files are copied/symlinked to? After a successful compile, all new files in the temp directly could be copied back to the source dir, so even third party software output shows up. If existing files were overwritten/changed, the user could be warned. An additional advantage would be that on a failed compile, the intermediate files would not clutter the source dir.

Alternatively, something like OverlayFS might be an option.

[cscheid]

Maybe quarto could do the processing in a separate temp dir, where all the source files are copied/symlinked to?

This breaks most code in R/Python that uses the filesystem.

Alternatively, something like OverlayFS might be an option.

The majority of our users is on Windows.

[StianBerger]

I want to deploy Quarto in a server environment, with multiple users and possibly concurrent workloads.
I fear that concurrent renderers might mix up the pipeline and leak information across rendered reports (or just crash?).
It is also quite bad practice to give an application write access to folder with run-able code.

I am considering to create a random tmp folders for each request, where I would have to copy all dependencies, and run execution there. That way i can safely give quarto write access, and I don't have to fear cross contamination. But it is a bit cumbersome.

A parameter for tmp files would be welcome.

[cscheid]

It is also quite bad practice to give an application write access to folder with run-able code.

That's unfortunately not a workable solution in Quarto, where downstream tools (knitr, latex, and most user's own code cells) expect files to be created in those directories.

I am considering to create a random tmp folders for each request, where I would have to copy all dependencies, and run execution there. That way i can safely give quarto write access, and I don't have to fear cross contamination. But it is a bit cumbersome.

That's probably your best bet.

[mcanouil]

Seems to be a duplicate (in some aspects) of the most recent:

• Configure location of intermediate md files generated from qmd #9299