feat: added llms.txt and robots.txt#1145
Conversation
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Greptile Summary
This PR adds two important files for web crawling and AI indexing standards to the Sim platform. The changes implement a robots.txt file in the public directory and an llms.txt API route at /llms.txt/route.ts.
The robots.txt file follows standard web conventions by strategically controlling crawler access - it allows modern AI crawlers (GPTBot, Claude-Web, ChatGPT-User, CCBot, anthropic-ai, Bytespider) full access to the site while restricting traditional search engine crawlers from sensitive areas like /api/, /workspace/, and /chat/. This approach balances SEO discoverability with user privacy protection.
The llms.txt route implements an emerging standard for helping AI systems understand platform capabilities. It serves a plain text description of Sim's AI agent workflow builder, including key features, use cases, and resources. The route returns static content with appropriate caching headers (Cache-Control: public, max-age=86400) for performance optimization. The content covers Sim's core value proposition as a visual platform for building AI agent workflows, lists various use cases from chatbots to financial planning agents, and provides links to documentation and GitHub.
These additions integrate well with the existing codebase structure, following Next.js API route conventions and placing static assets in the standard public directory. The implementation supports the platform's goal of being discoverable by AI systems while maintaining control over what areas crawlers can access.
Confidence score: 4/5
- This PR is generally safe to merge with some minor content corrections needed
- Score reflects straightforward implementation with standard web practices, but lowered due to content quality issues that should be addressed
- Pay close attention to the llms.txt content for typos and duplicate entries before merging
2 files reviewed, 4 comments
| ## Use Cases | ||
| - AI Agent Workflow Automation | ||
| - RAG Agents | ||
| - RAG Systesm and Pipline |
There was a problem hiding this comment.
syntax: Typo: 'Systesm and Pipline' should be 'Systems and Pipeline'
| - RAG Systesm and Pipline | |
| - RAG Systems and Pipeline |
| - Marketing Agent Workflows | ||
| - Deep Research Workflows | ||
| - Marketing Agent Workflows |
There was a problem hiding this comment.
syntax: 'Marketing Agent Workflows' is listed twice - remove the duplicate on line 22
| - Marketing Agent Workflows | |
| - Deep Research Workflows | |
| - Marketing Agent Workflows | |
| - Marketing Agent Workflows | |
| - Deep Research Workflows |
| - [GitHub](https://github.com/simstudioai/sim) | ||
|
|
||
| ## Resources | ||
| - [GitHub](https://github.com/simstudioai/sim)` |
There was a problem hiding this comment.
style: GitHub link is duplicated in both 'Getting Started' and 'Resources' sections - consider removing one or differentiating their purposes
| Disallow: /workspace/ | ||
| Disallow: /chat/ | ||
| Disallow: /.well-known/ | ||
| Disallow: /invite/ No newline at end of file |
There was a problem hiding this comment.
style: Missing newline at end of file. Add a newline character after the last Disallow directive.
…is for queueing docs in kb (#1143) * improvement(kb): created knowledge, chunks, tags services and use redis for queueing docs in kb * moved directories around * cleanup * bulk create docuemnt records after upload is completed * fix(copilot): send api key to sim agent (#1142) * Fix api key auth * Lint * ack PR comments * added sort by functionality for headers in kb table * updated * test fallback from redis, fix styling * cleanup copilot, fixed tooltips * feat: local auto layout (#1144) * feat: added llms.txt and robots.txt (#1145) * fix(condition-block): edges not following blocks, duplicate issues (#1146) * fix(condition-block): edges not following blocks, duplicate issues * add subblock update to setActiveWorkflow * Update apps/sim/app/workspace/[workspaceId]/w/[workflowId]/components/workflow-block/components/sub-block/components/condition-input.tsx Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix dependency array * fix(copilot-cleanup): support azure blob upload in copilot, remove dead code & consolidate other copilot files (#1147) * cleanup * support azure blob image upload * imports cleanup * PR comments * ack PR comments * fix key validation * improvement(forwarding+excel): added forwarding and improve excel read (#1136) * added forwarding for outlook * lint * improved excel sheet read * addressed greptile * fixed bodytext getting truncated * fixed any type * added html func --------- Co-authored-by: Adam Gough <adamgough@Mac.attlocal.net> * revert agent const * update docs --------- Co-authored-by: Siddharth Ganesan <33737564+Sg312@users.noreply.github.com> Co-authored-by: Emir Karabeg <78010029+emir-karabeg@users.noreply.github.com> Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> Co-authored-by: Adam Gough <77861281+aadamgough@users.noreply.github.com> Co-authored-by: Adam Gough <adamgough@Mac.attlocal.net>
…is for queueing docs in kb (simstudioai#1143) * improvement(kb): created knowledge, chunks, tags services and use redis for queueing docs in kb * moved directories around * cleanup * bulk create docuemnt records after upload is completed * fix(copilot): send api key to sim agent (simstudioai#1142) * Fix api key auth * Lint * ack PR comments * added sort by functionality for headers in kb table * updated * test fallback from redis, fix styling * cleanup copilot, fixed tooltips * feat: local auto layout (simstudioai#1144) * feat: added llms.txt and robots.txt (simstudioai#1145) * fix(condition-block): edges not following blocks, duplicate issues (simstudioai#1146) * fix(condition-block): edges not following blocks, duplicate issues * add subblock update to setActiveWorkflow * Update apps/sim/app/workspace/[workspaceId]/w/[workflowId]/components/workflow-block/components/sub-block/components/condition-input.tsx Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix dependency array * fix(copilot-cleanup): support azure blob upload in copilot, remove dead code & consolidate other copilot files (simstudioai#1147) * cleanup * support azure blob image upload * imports cleanup * PR comments * ack PR comments * fix key validation * improvement(forwarding+excel): added forwarding and improve excel read (simstudioai#1136) * added forwarding for outlook * lint * improved excel sheet read * addressed greptile * fixed bodytext getting truncated * fixed any type * added html func --------- Co-authored-by: Adam Gough <adamgough@Mac.attlocal.net> * revert agent const * update docs --------- Co-authored-by: Siddharth Ganesan <33737564+Sg312@users.noreply.github.com> Co-authored-by: Emir Karabeg <78010029+emir-karabeg@users.noreply.github.com> Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> Co-authored-by: Adam Gough <77861281+aadamgough@users.noreply.github.com> Co-authored-by: Adam Gough <adamgough@Mac.attlocal.net>
1 participant
Summary
Add robots.txt and llms.txt
Type of Change
Checklist