llms.txt PHP

This library supports you in creating, reading, and validating llms.txt Markdown files via PHP. It targets v2 of the specification. A good example llms.txt file is the one from the uv project.

What's llms.txt?

Think of it like robots.txt for LLMs. The evolving spec is available over here, its second version is summarised here. For the structure of a llms.txt file you can also have a look at this repository's llms.txt file.

Installation and usage

composer require stolt/llms-txt-php

Creating a llms.txt file

use Stolt\LlmsTxt\LlmsTxt;
use Stolt\LlmsTxt\Section;
use Stolt\LlmsTxt\Section\Link;

$section1 = (new Section())->name('Section name')
    ->addLink((new Link())->urlTitle('Link title')
        ->url('https://link_url')->urlDetails('Optional link details')
    );
$section2 = (new Section())->name('Optional')
    ->link((new Link())->urlTitle('Link title')
        ->url('https://link_url')
    );

$llmsTxt = (new LlmsTxt())->title('Test title')
  ->description('Test description')
  ->details('Test details')
  ->addSection($section1) // OR ->addSections([$section1, $section2])
  ->section($section2) // alias method
  ->toString(); // OR ->toFile('/path/to/llmsTxtToBe.md');

Setting and reading the Optional section

Since v2 of the specification the Optional section no longer carries mechanical semantics, it stays a convention for secondary links. The optional and getOptional methods give direct access to it, without having to know the section name.

use Stolt\LlmsTxt\LlmsTxt;
use Stolt\LlmsTxt\Section;
use Stolt\LlmsTxt\Section\Link;

$llmsTxt = (new LlmsTxt())->title('Test title')
    ->description('Test description')
    ->details('Test details')
    ->optional((new Section())->addLink(
        (new Link())->urlTitle('Secondary link')->url('https://link_url')
    ));

$optionalSection = $llmsTxt->getOptional(); // null when there is none

The section is named Optional automatically, and an already present Optional section is replaced. The accessor also works on a parsed llms.txt file.

Making a llms.txt file discoverable

Version two of the specification introduced link relations so agents don't have to guess where things are. rel="alternate" type="text/markdown" points at the Markdown version of a page, rel="describedby" at the llms.txt file covering it. Both can be served as HTML <link> elements or as an HTTP Link header.

use Stolt\LlmsTxt\Discovery;

$discovery = new Discovery();

$discovery->asLinkElements('/docs/llms.txt', '/docs/page.html.md');
$discovery->asLinkHeader('/docs/llms.txt', '/docs/page.html.md');

Value of the rendered link elements, and of the header value, which excludes the Link: field name so it can be handed straight to header():

<link rel="alternate" type="text/markdown" href="/docs/page.html.md">
<link rel="describedby" href="/docs/llms.txt">

</docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"

The same relations can be read back out of HTML content or a Link header value.

$discovery->describedByUrls($html);              // ['/docs/llms.txt']
$discovery->markdownAlternates($html);           // ['/docs/page.html.md']

$discovery->describedByUrlsFromHeader($header);
$discovery->markdownAlternatesFromHeader($header);

Two URL rules of the specification are available as helpers. A page may offer its Markdown version either by appending .md or by swapping its extension for .md, and a llms.txt file covers the pages under its path, where the most specific file applies.

$discovery->markdownUrls('/docs/page.html');  // ['/docs/page.html.md', '/docs/page.md']
$discovery->markdownUrls('/docs/');           // ['/docs/index.html.md', '/docs/index.md']
$discovery->markdownUrls('/docs/page.md');    // ['/docs/page.md'], already a Markdown URL

$discovery->coveringUrls('/docs/a/b.html');   // ['/docs/a/llms.txt', '/docs/llms.txt', '/llms.txt']
$discovery->coveringUrl('/docs/a/b.html', ['/llms.txt', '/docs/llms.txt']); // '/docs/llms.txt'

Combining Discovery and LlmsTxt

Discovery locates files, LlmsTxt reads them, which makes the two complement each other. An agent that has the HTML of a page, or its Link header, can follow the describedby relation and parse what it finds.

use Stolt\LlmsTxt\Discovery;
use Stolt\LlmsTxt\LlmsTxt;

$discovery = new Discovery();

foreach ($discovery->describedByUrls($html) as $llmsTxtUrl) { // OR describedByUrlsFromHeader($header)
    $llmsTxt = (new LlmsTxt())->parse($llmsTxtUrl);

    if ($llmsTxt->validate()) {
        $sections = $llmsTxt->getSections();
        // ...
    }
}

The other direction, when serving a page, is to announce the llms.txt file covering it next to its Markdown version. The keys of the file list are the URLs the llms.txt files are served under, and the most specific one covering the requested page wins.

$llmsTxts = [
    '/llms.txt' => (new LlmsTxt())->parse('/path/to/llms.txt'),
    '/docs/api/llms.txt' => (new LlmsTxt())->parse('/path/to/docs/api/llms.txt'),
];

$page = '/docs/api/v2/endpoints.html';

\header('Link: ' . $discovery->asLinkHeader(
    $discovery->coveringUrl($page, \array_keys($llmsTxts)) ?? '',
    $discovery->markdownUrls($page)[0]
));

Value of the sent header:

Link: </docs/api/v2/endpoints.html.md>; rel="alternate"; type="text/markdown", </docs/api/llms.txt>; rel="describedby"

Since the markdownUrls method returns the URL forms a Markdown version may be served under, it also composes with the $fetcher of the context expansion below. The following fetcher resolves every linked page through its Markdown version and falls back to the page itself when there is none, which keeps HTML out of the context file.

use Stolt\LlmsTxt\Discovery;
use Stolt\LlmsTxt\LlmContext;

$discovery = new Discovery();
$llmContext = new LlmContext();

$fetcher = static function (string $url) use ($discovery, $llmContext): string {
    foreach ($discovery->markdownUrls($url) as $markdownUrl) {
        try {
            return $llmContext->fetch($markdownUrl);
        } catch (\RuntimeException) {
            continue;
        }
    }

    return $llmContext->fetch($url);
};

$context = (new LlmsTxt())->parse('/path/to/llmsTxt.md')->toLlmContext(false, $fetcher);

All of the above is covered by DiscoveryIntegrationTest.php.

Expanding a llms.txt file into a LLM context file

Linked documents can be expanded into an XML context file. Version two of the specification dropped its llms_txt2ctx context expansion tooling, and with it the mechanical meaning of the Optional section, so all sections are expanded by default. Pass $skipOptional to leave the secondary links out and get a shorter context.

use Stolt\LlmsTxt\LlmsTxt;

$llmsTxt = (new LlmsTxt())->parse('/path/to/llmsTxt.md'); // OR parse('markdown-string')

$context = $llmsTxt->toLlmContext(); // OR ->toLlmContext(true) to skip the Optional section
$llmsTxt->toLlmContextFile('/path/to/llm-ctx.xml');

Tip

Pass a $fetcher callable (string $url): string to resolve URLs without hitting the network (for tests or a custom HTTP client). The default fetcher reads local files and otherwise uses PHP streams.

The context file is a well-formed XML, the details of the llms.txt file and the fetched documents are escaped as character data. Quotes are kept as they are, since they carry no meaning outside an attribute value.

Expanding a llms.txt file into a llms-full.txt file

The same linked documents can be expanded into a llms-full.txt file, the Markdown counterpart of the XML context file. It keeps the header of the llms.txt file, and turns every file list entry into a ### Link title block holding its details, its source URL, and the fetched document.

use Stolt\LlmsTxt\LlmsTxt;

$llmsTxt = (new LlmsTxt())->parse('/path/to/llmsTxt.md'); // OR parse('markdown-string')

$full = $llmsTxt->toFull(); // OR ->toFull(true) to skip the Optional section
$llmsTxt->toFullFile('/path/to/llms-full.txt');

Value of $full:

# Title

> Optional description goes here

Optional details go here

## Section name

### Link title

Optional link details

Source: https://link_url

<!-- the fetched document -->

Tip

toFull and toFullFile take the same $skipOptional flag and $fetcher callable as the context expansion above. The fetched documents are inlined as they are, since a llms-full.txt file is meant to carry their full text.

Validating and reading a llms.txt file and its parts

The title is the only element the specification requires, so a missing one is the only validation error. The description, details, and file lists are recommended, missing ones are reported as validation warnings.

use Stolt\LlmsTxt\LlmsTxt;

$llmsText = (new LlmsTxt())->parse('/path/to/llmsTxt.md'); // OR parse('markdown-string')

if ($llmsText->validate()) {
    $title = $llmsText->getTitle();
    $description = $llmsText->getDescription();
    $details = $llmsText->getDetails();
    $sections = $llmsText->getSections();
}

In case you want to get the exact validation errors, you need to call validate with the detailed flag sat to true and then use the errors() method like shown below. The recommended, but not required, elements a file is missing are available via the warnings() method.

use Stolt\LlmsTxt\LlmsTxt;

$llmsText = (new LlmsTxt())->parse('/path/to/llmsTxt.md'); // OR parse('markdown-string')

$validationResult = $llmsTxt->validate(true);
if ($validationResult->isValid()) {
    $title = $llmsText->getTitle();
    $description = $llmsText->getDescription();
    $details = $llmsText->getDetails();
    $sections = $llmsText->getSections();
} else {
    $validationErrors = $validationResult->errors();
    // ...
}

if ($validationResult->hasWarnings()) {
    $validationWarnings = $validationResult->warnings();
    // ...
}

Tip

To interact with llms.txt files from the console, the complement package llms-txt-php-cli might come in handy. The complementary package also includes four AI skills that can be used to interact with llms.txt files.

Inline LLM instructions in HTML

Vercel proposed a non-formal standard for inlining LLM instructions in HTML, based on the llms.txt standard.

use Stolt\LlmsTxt\LlmsTxt;
use Stolt\LlmsTxt\Section;
use Stolt\LlmsTxt\Section\Link;

$section1 = (new Section())->name('Section name')
    ->addLink((new Link())->urlTitle('Link title')
        ->url('https://link_url')->urlDetails('Optional link details')
    );
$section2 = (new Section())->name('Optional')
    ->link((new Link())->urlTitle('Link title')
        ->url('https://link_url')
    );

$llmsTxtContent = (new LlmsTxt())->title('Test title')
  ->description('Test description')
  ->details('Test details')
  ->sections([$section1, $section2])
  ->asScriptTag(); // OR ->toEmbeddedInScriptTag()

Value of $llmsTxtContent:

<script type="text/llms.txt">
<!-- programmatically assembled llms.txt content -->
</script>

For more usage examples, have a look at the tests i.e. LlmsTxtTest.php.

Extract LLM instructions from HTML

use Stolt\LlmsTxt\Extractor;

$html = <<<HTML
<html>
  <body>
    <script type="text/llms.txt"># first llms.txt content</script>
    Some other content.
    <p>And some more content.</p>
    <br />
    <script type="text/llms.txt"># second llms.txt content</script>
  </body>
</html>
HTML;

$llmsTxts = (new Extractor())->extractFromHtml($html); // OR ->extractFromFile('/path/to/file.html')

Value of $llmsTxts:

array(2) {
  [0]=>
  string(11) "# first llms.txt content"
  [1]=>
  string(12) "# second llms.txt content"
}

To retrieve already parsed llms.txt object instances, pass the parse flag to the available extraction methods.

Value of $llmsTxts when parsed:

array(2) {
  [0]=>
  object(Stolt\LlmsTxt\LlmsTxt)#11 (5) {
    ["hasBeenParsed":"Stolt\LlmsTxt\LlmsTxt":private]=>
    bool(true)
    ["title":"Stolt\LlmsTxt\LlmsTxt":private]=>
    string(22) "first llms.txt content"
    ["description":"Stolt\LlmsTxt\LlmsTxt":private]=>
    string(0) ""
    ["details":"Stolt\LlmsTxt\LlmsTxt":private]=>
    string(0) ""
    ["sections":"Stolt\LlmsTxt\LlmsTxt":private]=>
    array(0) {
    }
  }
  [1]=> // ... ommitted for brevity
}

Running tests

composer test

License

This library is licensed under the MIT license. Please see LICENSE.md for more details.

Changelog

Please see CHANGELOG.md for more details.

Contributing

Please see CONTRIBUTING.md for more details.