Suggestion: /.well-known/llms.txt instead

[simonw]

These days there's a increasingly adopted alternative to the /robots.txt and /sitemap.xml and /favicon.ico standards - the /.well-known prefix: https://en.wikipedia.org/wiki/Well-known_URI

These have been registered with IANA since ~2010: https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml

The RFC has the argument for why this is a good idea, which honestly I haven't read in a few years so I'm a bit rusty on: https://datatracker.ietf.org/doc/html/rfc5785

Oh, maybe this is it (quoting the RFC):

When this happens, it is common to designate a "well-known location" for such data, so that it can be easily located. However, this approach has the drawback of risking collisions, both with other such designated "well-known locations" and with pre-existing resources.

To address this, this memo defines a path prefix in HTTP(S) URIs for these "well-known locations", "/.well-known/". Future specifications that need to define a resource for such site-wide metadata can register their use to avoid collisions and minimise impingement upon sites' URI space.

[3052]

100%. its kind of sad that a group of this size didn't bother doing any research on prior art before throwing this up.

[martinthomson]

A more complete argument can be found in RFC 8820. The original name of that document was a variation on "get off my lawn", which is a good summary of the general problem here.

[ShaneCurcuru]

Came here to also ask that this be proposed as well known, per: https://datatracker.ietf.org/doc/html/rfc8615

[jeffbryner]

100% Here's examples of all the other .well-known's in use https://en.m.wikipedia.org/wiki/Well-known_URI#List_of_well-known_URIs

[NatoBoram]

/.well-known/llms.md would make more sense since it's a markdown file

[jfunction]

I agree. Looks like there are some hanging questions. Will we get a decision on this? Who decides, and who should be assigned?

[simonwfarrow]

I created specification for adding llms.txt and llms-full.txt to well-known-uris. PR here
I reached out on Discord, and I think the author (@jph00) responded to say happy for someone to give it a go.

Next step is to submit the above spec as a provisional listing - happy to do this as long as everyone agrees.

[jph00]

Go for it @simonwfarrow :)

[kapouer]

I propose that we all do a delayed redirection from /llms.txt to /.well-known/llms.md
This will probably have more effect than talking nicely.

[janwilmake]

genius

[jph00]

its kind of sad that a group of this size didn't bother doing any research on prior art before throwing this up

"a group of this size" is just me. I did not have the resources to go through the RFC-required registration process before creating a new .well-known resource, and also doing so would make it impossible to go to the next planned step, of allowing an llms.txt file to be placed at sub-path level.

As I've mentioned before, I'm happy for folks to go through the registration process if they wish, and there is currently an open PR with some changes required before it can be accepted (since it's not compatible with the current proposal). I don't think it's a great idea to go down this path, personally, since we'd then end up with different file names in the root vs sub-paths, but I'm not in charge of the internet so I don't get to tell anyone what to do!