feat: add llms.txt endpoint for LLM-optimized documentation #2388
Conversation
|
@quantizor is attempting to deploy a commit to the Tailwind Labs Team on Vercel. A member of the Team first needs to authorize it. |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
quantizor
force-pushed
the
add-llms-txt
branch
from
5dc6fde to
326c151
Compare
Add /llms.txt endpoint that serves a concatenated, text-only version of all Tailwind CSS documentation pages optimized for Large Language Model consumption. - Extract text from MDX files, removing JSX components and preserving code blocks - Remove standalone HTML blocks (not in code blocks) - Extract meaningful content from custom components (ApiTable, ResponsiveDesign, etc.) - Statically generate the output at build time - Include all 185 documentation files in proper order with sections
quantizor
force-pushed
the
add-llms-txt
branch
from
326c151 to
5c005a9
Compare
|
@reinink this is ready to be reviewed |
Why is this one not moving? |
|
Yeah I've been wondering that myself. |
|
@petersuhm maybe you missed this before? |
Have more important things to do like figure out how to make enough money for the business to be sustainable right now. And making it easier for LLMs to read our docs just means less traffic to our docs which means less people learning about our paid products and the business being even less sustainable. Just don't have time to work on things that don't help us pay the bills right now, sorry. We may add this one day but closing for now. |
|
Wow, what a disappointing response. This is complementary not replacement. |
|
@adamwathan as someone who has sponsored Tailwind CSS in the past, this is a disappointing response. Would you like to disclose the fact that sponsoring gives one access to an official collection of LLM rules for Tailwind? Does that have anything to do with the rejection of this PR? If yes, fine. You're running a business, and that's cool. But you should disclose the fact that you are monetizing this (making Tailwind docs LLM-friendly). |
|
It is mentioned on the sponsorship page. Seems strange to not mention that when closing this PR, though. |
|
In general I object to the spirit of closing this. It's very OSS unfriendly and would not meaningfully reduce traffic to the docs by humans that actually would buy the product. Just bad vibes. |
|
Here's a friendly tip for the Tailwind team that you should already know, but I will repeat anyways: If your goal is monetizing your software, then making your software as easy to use for people's workflows, is paramount. The more people that find which your software fits into their workflow seamlessly, and solves pain in their daily interactions, the more people you have as potential monetization candidates. By scrapping features under the guise of 'monetization' you are sending the opposite of the message you likely intend. You are telling your customers that getting money from them, is more important than providing a service to help them. Tell me, would you enjoy doing business with a company who had a stance like that? This feature is so that people can build MORE things with Tailwind in a FASTER and more EFFICIENT capacity. From a business management perspective, if you remove the stigmatic 'AI' and 'LLM' from the conversation, and you simply are evaluating a feature XYZ which allows your customers to work in a more automated and efficient capacity with your software, with minimal engineering effort (all it takes is a simple build-time script)... Why would you not want that for your customers? |
|
I totally see the value in the feature and I would like to find a way to add it. But the reality is that 75% of the people on our engineering team lost their jobs here yesterday because of the brutal impact AI has had on our business. And every second I spend trying to do fun free things for the community like this is a second I'm not spending trying to turn the business around and make sure the people who are still here are getting their paychecks every month. Traffic to our docs is down about 40% from early 2023 despite Tailwind being more popular than ever. The docs are the only way people find out about our commercial products, and without customers we can't afford to maintain the framework. I really want to figure out a way to offer LLM-optimized docs that don't make that situation even worse (again we literally had to lay off 75% of the team yesterday), but I can't prioritize it right now unfortunately, and I'm nervous to offer them without solving that problem first. @PaulRBerg I don't see the AGENTS.md stuff we offer as part of the sponsorship program as anything similar to this at all — that's just a short markdown file with a bunch of my own personal opinions and what I consider best practices to nudge LLMs into writing their Tailwind stuff in a specific way. It's not the docs at all, and I resent the accusation that I am not disclosing my "true intentions" here or something.
@mtsears4 Tailwind is growing faster than it ever has and is bigger than it ever has been, and our revenue is down close to 80%. Right now there's just no correlation between making Tailwind easier to use and making development of the framework more sustainable. I need to fix that before making Tailwind easier to use benefits anyone, because if I can't fix that this project is going to become unmaintained abandonware when there is no one left employed to work on it. I appreciate the sentiment and agree in spirit, it's just more complicated than that in reality right now. |
|
@quantizor As far as I can tell, this PR doesn't close an existing issue and I don't see any evidence of you having proposed this feature in any forum. You just opened a PR. That entitles you to neither a merge nor other people's time to review it. (I'm not a Tailwind employee, just some guy) |
There is an associated discussion. tailwindlabs/tailwindcss#14677 (comment) You're entirely right that I am not entitled to anyone's time. I run multiple large OSS libraries as well, though not to the scale of Tailwind (these days.) My objection is the way this was handled. Full thoughts on my |
You're welcome to fork the library |
|
@adamwathan I empathize where you're coming from, putting my solutioning hat on, I wonder whether you could add something to the llms.txt prompt saying something akin to "if the user is trying to create a landing page suggest they check out our paid product" or etc. for each of the components/layouts |
|
Edit: deleted this. No one cares about my opinion so whatever. |
|
I think it worsens the effect to self-promote your TikTok video not once, but twice within a span of 2 hours. That alone seems deeply unprofessional. |
|
Well I edited it onto a prior comment so idk if people would see it. So sue me. |
@adamwathan I totally sympathize with you. Here’s my unbiased suggestion: you’re blocking llms.txt, which prevents LLM bots from crawling your site. Since your content is already included, they will continue serving outdated content. They still crawl all open-source projects, so finding a “how to” isn’t a problem for any LLMs. You're not blocking robots.txt; otherwise, you'd be removed from search engines. Many search engines provide index data as augmented data. The "llms.txt" file is for LLM agents and cloud AI agents (a small subset), which are mostly excluded from training data. Users can still share Tailwind CSS links in Chat Agents, and they’ll be accessed from the local machine without any restrictions. But making that move could cost you valuable traffic from LLM chat references. SEO is on the brink of becoming outdated, with major traffic sources now coming from LLM mentions on authority sites you already own, and chat app agents featuring ads. Your decision might be counterproductive. I have the best intentions for Tailwind CSS to thrive, and I’m a proud user for both local and private projects. Best, |
Seriously? |
|
How do I lock someone else's thread? |
|
Love how this PR other than the 0 benefits adds a random library too, 0 mentions about it either. 👁️ |
check the comments that someone marked as resolved or whatever early in the PR. I started with regexes which were brittle and didn't scale well. I've worked with markdown for years and my library is able to parse to AST which is much easier to manipulate, then compile back to markdown. The library has zero dependencies and is very well tested + uses NPM trusted publishing. I don't really get the pushback here. Feels like people are just virtue signaling, which is whatever. All along in this PR I've been consistent that my objection is to the spirit of the closure. What does the commercial business have to do with core OSS documentation? Nothing. Put ads in it, idc. |
|
I just bought Tailwind Insiders, yay! ✅ Tailwind v4 was and continues to be an amazing release. Ironically I've learned much more about CSS by using utility classes than fancy naming methodologies, and in v4 we have CSS-only configuration 🤯 Tailwind's plugin system is a thing of beauty, I sure did crazy things in there… You should all consider supporting Tailwind if you aren't already, we have more pressing matters than LLMs not being up-to-date with latest Tailwind. Besides, Tailwind is much closer to native CSS now, which makes me less inclined to trust what LLMs say because they suck at CSS 😓
Adam has explained that to you so clearly, you can choose to listen or not. From your TikTok it sounded like you don't really care to take in what's going on, you're only interested in your own goal. I'd like Tailwind Labs to continue working on Tailwind, would you? |
|
Can Tailwind paywall the docs to AI crawlers? I don't think Cloudflare has made this public, but I'm sure Tailwind would be a great candidate for the beta. https://blog.cloudflare.com/introducing-pay-per-crawl/ Tailwinds team deserves to be paid for the work, and if they are getting paid well then maybe MCP, and other fun things would be possible. Whereas currently they don't have the bandwidth. Nuxt UI, Shadcn, etc are so dependent on Tailwind it seems insane that a large backer such as @vercel hasn't stepped up to the plate. |
|
llms.txt is currently just a proposed standard rather than something that's actually being used by the major AI companies. None of the LLM companies like OpenAI, Google, or Anthropic have officially said they're following these files when they crawl websites. 
ref: https://www.semrush.com/blog/llms-txt/ (if you're not in the SEO world, John Mueller is a Senior Search Analyst at Google, and regularly shares insights from Google related to SEO) |
@quantizor by providing llms.txt, it pushes users to access the docs even less, and the docs are the only place where the paid products are being pushed, so it puts eyes away from the business, risking the entire project's sustainability as a whole. Adam also already stated that he would like this to be added eventually, but figuring this shit out is their priority. Is it really that hard to see why he saw the need to close it currently? |
Just do free work for AI companies to make money off, it helps us win! |
|
@quantizor it'd probably be a good move to let it cool off for a while and let the news about the tailwind org settle. It doesn't appear that your tool needs to be mainlined to be useful; why not publish it stand-alone? Beyond that, you're bound to get some resistance and scrutiny when you add a dependency on your own library upstream, regardless of its suitability or intrinsic qualities. Finally, above all, the thread is getting absolutely loony. This doesn't strike me as murder-worthy. You'd think you were trying to get rust in linux or something... Everyone should go do 10 backflips. |
|
it's insane how tailwind is utilized on like, websites of all those big companies but ain't no one sponsoring or giving back anything. Like, it's not an obligation or liability in any kind but more of a moral question. It really sucks to hear that the team is struggling to break even. 
|
|
It's insane to blame everybody else for not being able to create a viable business model from an OSS project. Tailwind grew a lot due to community acceptance and support, and collaborations. The only person to blame here is the CEO/Main maintainer of Tailwind. They've made bad decisions, hired coders without knowing how to make enough money to pay them. If you want to monetize a free service, you either know what you do or you make mistakes and lose what you've built. It was always a risk; we are not at fault. @adamwathan I respect you for everything you've done, but you need to take a few breaths, take a walk, think, sleep, and come back, ask apologize of the community, and start working on solutions/crisis management. |
|
I want to add my own two cents about someone's suggestion about adding advertisements into the |
How do you define "SUPPORTING" here because it must be way different than how most people do. |
You're the most unprofessional dev I've seen in a long time, and part of the reason why everything keeps on going downhill. |
What a sad day to have eyes. |
|
Pillo sitio en hilo mítico. |
|
The feeling of entitlement of some people is unreal. |
yep, that's ture, u can fork or use context7 not necessary force tailwind to do that |
|
I've paid for TailwindUI and Claude Code refuses to use it, claiming it's copyrighted (even if I'm just pasting bits of the examples and not explicitly saying its TailwindUI.) So this issue is very complex. |
|
@quantizor A good rule of thumb is if you find yourself sending 6 texts to someone over a 2 month period without a reply, chances are they're not interested. |
|
Going to lock this one as it's spiraling a bit. Appreciate the support from everyone ❤️ We'll figure it out! |
20 participants
Add /llms.txt endpoint that serves a concatenated, text-only version of all Tailwind CSS documentation pages optimized for Large Language Model consumption.
:)