Add llms.txt generation for AI-friendly article extraction#486
Open
Add llms.txt generation for AI-friendly article extraction#486
Conversation
Implements the llms.txt standard (https://llmstxt.org/) to make PolicyEngine research articles more token-efficient for AI consumption. - Add scripts/generate-llms-txt.ts that generates: - /llms.txt - Index with links to sections - /llms-full.txt - All articles combined - /llms-research-us.txt - US articles only - /llms-research-uk.txt - UK articles only - Replace verbose Plotly JSON charts with text summaries using figure captions - Transform iframes to [Interactive: description] placeholders - Support optional ai_summary field in posts.json for custom summaries - Integrate into build process (runs before vite build) - Add tsx as dev dependency for running TypeScript scripts Closes #485 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Build scripts appropriately use console.log for progress output. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the llms.txt standard to make PolicyEngine research articles more token-efficient for AI consumption. This follows the pattern used by Bun, Svelte, and other projects.
Changes
Add
scripts/generate-llms-txt.tsthat generates:/llms.txt- Index with links to sections (~3KB)/llms-full.txt- All articles combined (~2.5MB, down from ~10MB+ with raw Plotly JSON)/llms-research-us.txt- US articles only (~1.3MB)/llms-research-uk.txt- UK articles only (~1.1MB)Replace verbose Plotly JSON charts with text summaries using figure captions
Transform iframes to
[Interactive: description]placeholdersSupport optional
ai_summaryfield in posts.json for custom summariesIntegrate into build process (runs before vite build)
Generated files are gitignored (built on deploy)
Token savings
[Interactive: ...]Workflow for PRs adding new articles
ai_summaryfield to your post inposts.jsonfor a custom AI-friendly summary**Figure 1: Winners by income decile**) - the caption becomes the chart summaryTest plan
npm run generate-llms-txt- generates all 4 files[Chart: caption][Interactive: ...]npm run build- llms.txt generated before vite buildCloses #485
🤖 Generated with Claude Code