AI instruments are in every single place — from chatbots that reply buyer inquiries to language fashions that summarize every thing from documentation to authorized textual content. However for those who’ve ever requested a mannequin like ChatGPT to elucidate your web site, your product, or your API, the outcomes may not really feel fairly proper. The truth is, generally they’re method off. And no, that’s not your fault.
The disconnect between web sites and LLMs
Massive language fashions (LLMs) like ChatGPT, Claude, or Gemini are educated to grasp a variety of content material. However after they attempt to interpret your web site at runtime, that’s, when somebody is actively asking them a query, they run into a number of core issues:
- HTML is noisy. Navigation bars, cookie banners, modal popups, and analytics scripts litter the web page.
- Context home windows are restricted. Most web sites are too massive for an LLM to course of all of sudden.
- Vital particulars are unfold throughout a number of pages or hidden in tables, code blocks, or feedback.
- Markdown docs could exist, however the mannequin typically can’t find them, and even know they exist.
So, once you ask an AI instrument to “clarify what this firm does” or “summarize this library API”, it typically will get caught. It both skips key context or grabs the mistaken alerts from cluttered markup.
It’s not unhealthy intent; it’s a design limitation.
Why it’s not your search engine optimisation’s fault, both
You’ve in all probability invested effort and time into SEO. Perhaps your robots.txt and sitemap.xml are in place. You’ve bought meta tags, structured knowledge, and clear inside hyperlinks. Good, however LLMs don’t all the time work like Google.
Conventional search engine optimisation helps your web site get discovered. Nonetheless, it doesn’t assure that AI instruments will perceive what a human consumer would. That’s the place a brand new proposal is available in.
Meet llms.txt: A easy method to assist AI perceive your web site
A rising variety of builders and AI researchers are adopting a light-weight, human-readable normal known as llms.txt.
What’s llms.txt?
llms.txt is a plain Markdown file positioned on the root of your web site that gives language fashions with a abstract of your challenge and direct hyperlinks to scrub, LLM-readable variations of essential pages. It’s designed for inference-time use, serving to AI instruments rapidly perceive a web site’s construction, goal, and content material with out counting on cluttered HTML or metadata meant for search engines like google and yahoo.
What it does:
- Offers a brief abstract of your web site or challenge
- Hyperlinks to scrub, LLM-ready Markdown variations of key pages
- Helps AI instruments discover precisely what issues, with out parsing messy HTML
Is it extensively supported? Not but
Proper now, no main LLM supplier formally helps llms.txt. Instruments like GPTBot (OpenAI), Claude (Anthropic), and Google’s AI crawlers don’t reference or comply with it as a part of their crawling habits. Some corporations like Anthropic publish llms.txt recordsdata themselves, however there’s no proof that any crawler is actively utilizing them in retrieval or coaching.
Nonetheless, it’s a low-effort, no-risk addition that helps put together your web site for a future the place structured LLM entry turns into extra standardized. And LLM-facing instruments, and even your individual AI brokers, could make use of it in the present day.
Instance use circumstances:
- A dev library hyperlinks to .md-formatted API docs and utilization examples.
- A college web site highlights course descriptions and tutorial insurance policies.
- A private weblog presents a simplified timeline of key initiatives or subjects.
You management the content material and the construction. LLMs profit from curated, LLM-aware context. And customers asking questions on your web site get higher solutions.
Utilizing our Yoast search engine optimisation plugin?
In the event you’re already utilizing our Yoast search engine optimisation (free or Premium) plugin, producing a llms.txt file is straightforward. Simply allow the function in your settings, and the plugin will mechanically create and serve a whole llms.txt file in your web site. You possibly can view it anytime at yourdomain.com/llms.txt.
Get Yoast search engine optimisation Premium
Unlock highly effective search engine optimisation insights with our Premium plugin, together with superior content material options, AI optimization instruments, and real-time knowledge constructed for the subsequent technology of search.
An LLM-friendly net isn’t the identical as a Google-friendly net
This doesn’t change search engine optimisation. Consider llms.txt as a companion to robots.txt. It tells AI bots: “Right here’s the good things. Skip the noise.”
Sitemaps assist crawlers discover every thing. llms.txt tells LLMs what to focus on.
It’s particularly helpful for:
- Builders and open-source maintainers
- Product entrepreneurs seeking to cut back assist load
- Groups that need chatbots to tug solutions from docs, not guess
You don’t want a brand new CMS or tech stack
All this requires is creating two issues:
- A primary llms.txt file in Markdown
- Ideally, you’d even have Markdown variations (.html.md) of key pages included alongside the originals, with the identical URL plus .md added.
No new instruments, plugins, or frameworks wanted, though some ecosystems are already including assist.
Right here’s an instance of a file mechanically constructed by Yoast search engine optimisation, because it has an llms.txt generator in-built:
Generated by Yoast search engine optimisation v25.3, that is an llms.txt file, meant for consumption by LLMs. That is the [sitemap](https://everydayimtravelling.com/sitemap_index.xml) of this web site.
# everydayimtravelling.com: Tales from our travels
## Posts
- [Test video](https://everydayimtravelling.com/test-video/)
- [A Journey Through Portugal’s Wine Country: A Suggested Wine Tour Route](https://everydayimtravelling.com/a-wine-tour-through-portugal/)
- [Travel essentials for backpackers FAQ](https://everydayimtravelling.com/travel-essentials-for-backpackers-faq/)
## Pages
- [Checkout](https://everydayimtravelling.com/checkout/)
- [Contact us](https://everydayimtravelling.com/contact-us/)
- [How we started this blog](https://everydayimtravelling.com/pagina-harry-potter/)
- [My account](https://everydayimtravelling.com/my-account/)
- [Cart](https://everydayimtravelling.com/cart/)
## Classes
- [Europe](https://everydayimtravelling.com/class/europe/)
- [Asia](https://everydayimtravelling.com/class/asia/)
- [South America](https://everydayimtravelling.com/class/south-america/)
- [Food](https://everydayimtravelling.com/class/meals/)
- [Western Europe](https://everydayimtravelling.com/class/europe/west-europe/)
## Tags
- [Budget](https://everydayimtravelling.com/tag/price range/) 
Serving to AI provide help to
So, if AI is misinterpreting your web site, producing misguided summaries, or skipping vital content material, there’s a purpose, and it’s fixable.
It’s not all the time your copy. Not your design or your metadata. It’s simply that these language instruments want a bit steerage. Sooner or later, llms.txt might be the best way to offer it to them, and also you achieve this in your phrases.
Do you want assist creating an llms.txt file or changing your present content material to Markdown for LLMs? Yoast search engine optimisation can mechanically generate an llms.txt file for you.
