What Is llms.txt and Should You Use It?
llms.txt is a proposed plain-text Markdown file, placed at the root of a website, that lists your most useful pages for language models in a short, curated form. It was proposed in 2024 by Jeremy Howard and is described at llmstxt.org. It is an optional convention, not a web standard enforced by any search engine. This guide explains what it is, what evidence exists on whether it helps, and how to create and test one. It is part of our wider work on generative engine optimization.
What llms.txt is
The proposal describes a file at /llms.txt that gives brief background on a site and links to more detailed Markdown pages. The idea is to give a model a curated overview, because websites are full of navigation, scripts and duplicate pages that are hard to parse. The proposal states it is meant to coexist with existing standards: a sitemap lists all pages for search engines, while llms.txt offers a curated overview for language models. It is most used for software documentation, where coding agents follow it to find API references and tutorials.
What an llms.txt file looks like
The format is simple Markdown:
- An H1 with the site or project name (required).
- A short blockquote summary.
- Optional paragraphs with context.
- H2 sections containing lists of links, each with a short note, such as
[Pricing](https://example.com/pricing.md): how plans work.
An optional section can mark secondary links that can be skipped when context is short. Some sites also publish an llms-full.txt with the full text of key content in one file. That variant is a convention used by documentation platforms, not part of a search engine requirement.
llms.txt vs robots.txt vs sitemap.xml
| File | Purpose | Honoured by search engines? |
|---|---|---|
| robots.txt | Tells crawlers what they may fetch | Yes, widely, for crawlers that respect it |
| sitemap.xml | Lists URLs for discovery | Yes, by major search engines |
| llms.txt | Curated guide to content for language models | Not documented as a ranking or citation input by Google or OpenAI |
llms.txt does not block or permit access. To control crawlers use robots.txt. For example, OpenAI documents separate bots for search, training and user actions (OpenAI bots documentation).
Does llms.txt help SEO or AI citations?
There is no public evidence from the major engines that it does. Google states that AI Overviews and AI Mode need no additional technical requirements or special files beyond indexing and snippet eligibility (Google Search Central). We are not aware of OpenAI, Google or Perplexity documenting llms.txt as an input to ranking or citation. Some AI tools and coding agents do read it, particularly for documentation, so it may help those specific workflows. Anyone claiming it increases citations in ChatGPT or AI Overviews is going beyond the available evidence.
That makes it a low-cost, low-certainty addition. If your site is a documentation or developer resource, it is more likely to be useful. For a typical service or ecommerce site, put effort into indexable pages, clear answers and authority first. See our guide to Google AI Overviews and GEO vs SEO for where the effort pays.
How to create an llms.txt file
- List the 10 to 30 pages that best explain what you do: services, pricing approach, key guides, documentation, contact and policies.
- Write a one-sentence note for each link that says what the reader will find.
- Group links under clear H2 headings and add a two-sentence summary at the top.
- Save it as plain text named llms.txt in the site root so it loads at yourdomain.com/llms.txt, served as text with a 200 status.
- Run the file through our llms.txt checker to validate the structure and links.
- Review it when you launch or retire important pages, and remove any URL that redirects or returns an error.
Mistakes to avoid
- Listing every URL. The point is curation; use the sitemap for completeness.
- Linking to pages that are noindexed, redirected or blocked, which sends mixed signals.
- Putting claims in the file that are not on the visible pages. Keep it consistent with your site.
- Expecting traffic from it. Measure instead; see how to measure AI visibility.
Should you add one?
Add it if you maintain documentation, publish many guides, or simply want a clean machine-readable summary and can maintain it. Skip it if it would pull time from fixing crawl, content or authority problems. Either way, check your robots.txt first, and use our free AI visibility audit for a broader view. If you want help prioritising, contact us or see our SEO services.
Frequently asked questions
What is an llms.txt file?
It is a proposed Markdown file at the root of a website that gives language models a short, curated overview of the site with links to its most useful pages.
Does llms.txt improve rankings in Google?
Google does not list it as a ranking factor and says AI Overviews and AI Mode need no special files. There is no documented ranking benefit.
Will llms.txt get me cited by ChatGPT or Perplexity?
No vendor documents that. Some tools and agents read it, but citation is not guaranteed and depends on many other factors.
Is llms.txt the same as robots.txt?
No. robots.txt controls which crawlers may fetch pages. llms.txt only suggests useful content and does not block or allow anything.
Where do I put the file and how do I test it?
Place it at yourdomain.com/llms.txt and validate it with the llms.txt checker to confirm the structure and that links return a 200 status.
Recent Articles
