Flag to produce easily consumable LLM output

[gshotwell]

Description

A growing number of sites are hosting a directory of markdown files which are easily consumed by LLMs. It seems like this would be a trivial thing for quarto do do because it would basically involve dumping the md files into a directory prior to rendering them.

The idea here is to have a website flag which would cause quarto to generate a reasonable llms.txt file.

https://llmstxt.org/

[cscheid]

I really like this idea!

I've long wanted to have output formats that include a full copy of the input .qmd (for rubs hands reasons :)) in the output. But this is simpler and more immediately useful.

[cscheid]

Let me hash out how we would go about implementing it. A few issues:

• The implementation @gshotwell suggested (just dumping the input .md) wouldn't actually work well because it would ignore the execution of Lua filters and website postprocessors.
• We would also need to make sure that the links work correctly. In this .md output, within-website links should be resolved to the .md output whenever possible, but not always (eg what happens if we link to a static HTML page in the website?).
• How should we resolve aliases?

I think that the most promising approach is to add a Lua filter to our chain that detects the llms.txt feature, embeds the post-filter AST into the HTML, and then a separate step creates the llm .md file from that embedded AST.

[gshotwell]

Right, I suppose the naive just dump the markdown approach might be good enough because most of the content would be there, but your approach sounds much better. If Quarto did a good job producing the raw markdown files, then the user could do post-processing steps to summarize those files to make them more efficient for llms to read.

One side benefit of all of this stuff is that it would immediately make all of Posit's package documentation way easier to navigate with AI tools.

@hamelsmu might be a good person to talk to about this as well.

[cscheid]

One side benefit of all of this stuff is that it would immediately make all of Posit's package documentation way easier to navigate with AI tools.

I agree with you on might rather than would. Is there something written on the degree to which LLMs do better in MD vs HTML? Presumably they would already know how to transform the latter into the former, right? So I assume this is about in part about context window size efficiency. Am I completely off base here?

[gshotwell]

I think it's two things:

1. Token efficiency: markdown just conveys way more information than html. For example if you inspect any of the quarto page source and ask "how many characters are about quarto here?" it's very small. Some LLM crawlers will parse all this stuff out, but they probably won't do that as well as quarto could.
2. Training data: since quite a lot of LLM training data is fed in as markdown, they tend to be a bit more fluent in it.