Providing a `llms.txt` file for better AI support.

[harshpreet931]

Description

can we make a llms.txt? llms are not that good in three.js for what I have felt like..

Solution

we need to do it!

Alternatives

It's kind of an additive feature

Additional context

No response

[Mugen87]

I have written a script for generating a llms.txt from the JSDoc here #30774 (comment) but the resulting file ends up too large, see #30774 (comment).

[harshpreet931]

Yeah... I saw Its huge.. is there something we can do to reduce or minify it? Something that can be under a million tokens... currently it is 1.001 mil tokens.. ideally can we have like 500-600k tokens?

[Mugen87]

I'm not sure about best practices in this area. One solution would be to just honor the core classes when generating llms.txt. That would save a lot of file size but also valuable information would be missing since many addons are frequently used and part of most three.js applications. Alternatively, we could reduce the markdown that is generated by removing e.g. the description text of methods and classes but that would also mean missing information for the AI.

IMO, the real issue are the size/token or context restrictions of AI tools which seem too strict to me. There are software projects with large code bases and they will face similar issues if they want to provide a llms.txt file.

[harshpreet931]

I agree that removing either addons or descriptions would mean losing useful context.

A couple of ideas we could explore:

• Split into multiple files (e.g. llms-core.txt, llms-addons.txt) so AI tools can load only what’s relevant instead of everything at once.
• Minify descriptions by keeping signatures, parameters, and return types, while shortening the natural language text.
• Configurable generation – allow contributors to choose between a full version and a lighter version when generating llms.txt.

This way we don’t lose too much information but can still stay under the token limits for current AI tools.

What do you think about trying a split-by-module approach first?

[Mugen87]

Split into multiple files (e.g. llms-core.txt, llms-addons.txt) so AI tools can load only what’s relevant instead of everything at once.

When you upload these files one-by-one into an AI tool, does a new upload overwrite the context from the previous upload?

[harshpreet931]

When you upload these files one-by-one into an AI tool, does a new upload overwrite the context from the previous upload?

The context from each uploaded file accumulates and persists throughout an active conversation. If the model goes out of context window then it truncates on FIFO basis.

[Makio64]

Making another repository like i did for animejs seems a good approach : https://github.com/Makio64/animejs-v4-ai-guidelines/

Im using this kind of approach for a better TSL as well.

[Mugen87]

Split into multiple files (e.g. llms-core.txt, llms-addons.txt) so AI tools can load only what’s relevant instead of everything at once.

@harshpreet931 I have updated the script so it's possible to control with a parameter the intended output.

node utils/ai/llms.mjs --mode=core 
node utils/ai/llms.mjs --mode=addons

You have to put below code into llms.mjs and place it into utils/ai/. You also need jsdoc-to-markdown as a dev dependency. Can you check if the resulting files llms-core.txt and llms-addons.txt are more usable? Instead of a single large file, the core is now 2.9MB and addons is 1MB. I'll attach them as a zip.

llms.zip

import fs from 'fs/promises';
import path from 'path';

import jsdoc2md from 'jsdoc-to-markdown';

// This script converts the API documentation from JSDoc to Markdown and writes everything
// into a single build/llms.txt output file.

const args = process.argv.slice( 2 );
const argsMap = new Map();

for ( const arg of args ) {

	const parts = arg.split( '=' );
	const [ name, value ] = parts;

	if ( name.startsWith( '--' ) ) {

		argsMap.set( name.substring( 2 ), value );

	}

}

build( argsMap );

async function build() {

	// gather all files potentially holding JS Doc

	const files = [];

	const config = JSON.parse( await fs.readFile( 'utils/docs/jsdoc.config.json', 'utf8' ) );

	const includes = config.source.include;
	const excludes = config.source.exclude;

	// process "mode" parameter

	const mode = argsMap.get( 'mode' );

	if ( mode === 'core' ) {

		excludes.push( 'examples/jsm' );

	} else if ( mode === 'addons' ) {

		excludes.push( 'src' );

	} else {

		throw new Error( 'Invalid mode: ' + mode );

	}

	//

	for ( const include of includes ) {

		const includeFiles = await toArray( readAllFiles( include ) ); // Replace toArray() with Array.fromAsync() when supported

		// filter out excluded and non-js files

		files.push( ... includeFiles.filter( function ( file ) {

			for ( const exclude of excludes ) {

				if ( file.startsWith( exclude ) === true || file.endsWith( '.js' ) === false ) {

					return false;

				}

			}

			return true;

		} ) );

	}

	// create output directory if not existing

	const outputDir = 'utils/ai/build';
	try {

		await fs.access( outputDir, fs.constants.R_OK | fs.constants.W_OK );

	} catch ( error ) {

		await fs.mkdir( outputDir );

	}

	// create output path and overwrite existing file

	const outputPath = path.join( outputDir, `llms-${mode}.txt` );
	await fs.writeFile( outputPath, '', { encoding: 'utf8', flag: 'w' } );

	// convert JSDoc to Markdown and save in output file

	await generateTitle( outputPath );
	await generateAPI( outputPath, files );

}

// helpers

async function generateTitle( outputPath ) {

	await fs.appendFile( outputPath, '# three.js \n\n' );
	await fs.appendFile( outputPath, '> JavaScript 3D Library. \n\n' );

}

async function generateAPI( outputPath, files ) {

	for ( const file of files ) {

		await fs.appendFile( outputPath, await jsdoc2md.render( { files: file } ) );

	}

}

async function* readAllFiles( dir ) {

	const files = await fs.readdir( dir, { withFileTypes: true } );

	for ( const file of files ) {

		if ( file.isDirectory() ) {

			yield* readAllFiles( path.join( dir, file.name ) );

		} else {

			yield path.join( dir, file.name );

		}

	}


}

async function toArray( asyncIterator ) {

	const arr = [];
	for await ( const i of asyncIterator ) arr.push( i );
	return arr;

}

[harshpreet931]

looks like a step in the right direction!

[aparajitaks]

Hi 👋, I’d like to work on this issue.
I’ve reviewed the discussion and implemented a version of the llms.mjs generator that splits output into llms-core.txt and llms-addons.txt, as discussed by @Mugen87 and @harshpreet931.
Could you please assign me this issue so I can raise a PR?