LLMs.txt for AutoGen Documentation Guides

[ekzhu]

For each page in documentation, we can compile a comparable llms.txt (or llms.md??) with markdown compilation of the content. The links in it should also reference corresponding links in llms.txt.  

Discussed in #6262

^{Originally posted by cc0zh April 10, 2025}
Hello AutoGen Team,
I hope this message finds you well.
I'd like to suggest adding an LLMs.txt file specifically for the AutoGen user guide documentation hosted at https://microsoft.github.io/autogen/stable/user-guide/.
An LLMs.txt file can provide valuable directives for large language models (LLMs) and AI agents interacting with the documentation. This guidance could include preferred crawl paths, sections suitable for summarization, disallowed areas, citation preferences, etc.

[LuluZhuu]

Hi @ekzhu , could I try working on this? Also, I would like to ask why this link https://microsoft.github.io/autogen/stable/user-guide/ is broken.

[ekzhu]

Hi @ekzhu , could I try working on this?

Sure! Could you please sketch out a plan how you would like to approach this?

Also, I would like to ask why this link https://microsoft.github.io/autogen/stable/user-guide/ is broken.

This is not a valid link. The full link to user guide is something like: https://microsoft.github.io/autogen/stable/user-guide/agentchat-user-guide/index.html

[LuluZhuu]

Hello, just to clarify, is this issue looking to automatically compile llm.txt files, or is it asking for a manually written llm.txt for each page?

Also, should there be 4 llm.txt files for the larger four sections of the user guide, like AgentChat or should there be more to cover the subsections on each page?

As for the llm.txt itself, I was thinking of formatting it like the attached sample.

sample_autogen_agentchat_llm.txt

Besides the previous questions, could you please give me some further guidance on how I should approach this issue? Thanks!

[ekzhu]

I think the automatic generation of llm.txt file can be done with tools like pandocs, or something off the shelf. Much of the doc is generated from Jupyter notebooks, so there should be some tools to convert Jupyter notebook JSON to markdown or text file.

Also, should there be 4 llm.txt files for the larger four sections of the user guide, like AgentChat or should there be more to cover the subsections on each page?

I am not sure what is the right size. Perhaps the simplest first step is to have one llm.txt for agentchat by concatenating all docs into one.

[ekzhu]

The generation should be a script that can be run locally and as part of the CI process. See how we build the docs as reference. Read about how docs are built in python/README.md.