llms.txt is a proposed standard: a Markdown file placed at /llms.txt that gives large language model tools a short, curated overview of your site with links to the pages worth reading. Jeremy Howard proposed it on 3 September 2024. It is not an official web standard, and Google Search does not use it.
What is llms.txt?
Websites are built for browsers. HTML full of menus, scripts and banners is hard for a model to read, and whole sites do not fit in a model’s context window. The proposal says a site owner can help by publishing one small file that says what the site is and where the clean, useful content lives, as plain Markdown versions of pages. The author expects it to be used mainly at inference time, when an AI tool helps a user, rather than for training.
It is a cousin of two older files, but does a different job:
| File | Audience | Job |
|---|---|---|
robots.txt |
Crawlers | Allow or block access (see robots.txt and noindex) |
sitemap.xml |
Search engines | List every URL you want indexed (see XML sitemaps) |
llms.txt |
AI tools and agents | A hand-picked reading list with short descriptions |
The file does not block or permit anything. To control AI crawlers, use robots.txt rules (see AI crawlers).
Does llms.txt help SEO or AI visibility?
For Google: no. Google’s guide to optimizing for generative AI features says you do not need to create new machine-readable files, AI text files, markup or Markdown to appear in Google Search. It says Google Search ignores such files, so they neither help nor harm. Creating them is fine for other services, it adds.
For other AI tools the picture is thin. The proposal is open and anyone can adopt it, and some developer tools and coding assistants read it for documentation sites. For general web search, I found no official statement from OpenAI, Anthropic or Perplexity that their search crawlers use llms.txt to choose sources. Check their current crawler documentation before you count on it (see ChatGPT search and Perplexity citations).
Impact is Low for that reason. The honest answer: write one if it takes 30 minutes and you have documentation, an API or a clear set of guides. Do not expect traffic from it.
What does the llms.txt format look like?
Per llmstxt.org, the file is Markdown with these parts in order:
- An H1 with the site or project name. It is the only required part.
- A blockquote with a short summary.
- Optional paragraphs or lists with more detail, but no headings.
- H2 sections that hold lists of links. Each entry is
[name](url), optionally followed by a colon and a note. - An Optional H2 section for links an AI tool can skip when it needs less context.
Like this:
# Acme Invoicing
> Acme Invoicing is accounting software for freelancers in the EU. It creates invoices, tracks payments and files VAT returns.
Prices are in euros. The API uses REST and JSON.
## Guides
- [Getting started](https://example.com/docs/start.md): create your first invoice in 5 minutes
- [VAT returns](https://example.com/docs/vat.md): how filing works per country
## API
- [API reference](https://example.com/docs/api.md): endpoints, auth and rate limits
## Optional
- [Changelog](https://example.com/changelog.md)The file can sit at the site root or in a subpath such as /docs/llms.txt, where it covers the URLs under that path.
How do you add llms.txt, step by step?
- Decide if it is worth it. Documentation, API, product help and developer content gain the most. A small brochure site gains little.
- List your 10 to 30 most useful pages. Pick the pages that answer real questions: pricing, setup, policies, main guides. Skip tag pages and thin posts.
- Make clean versions where you can. The proposal suggests Markdown copies of pages (like
page.html.md). Many docs platforms can export them. - Write the file in the format above. Keep the summary factual, with no marketing claims.
- Upload it to
/llms.txtat the root, served as plain text or Markdown with a 200 status. Open the URL in a browser to check. - Check the links and keep the file in sync when pages move. A generator in your build process avoids stale links.
- Do not reference it in robots.txt or the sitemap as if it were a ranking file. It has no role there.
- Check the logs after a month. Filter server or CDN logs for requests to
/llms.txtand see which user agents ask for it.
How do you measure the result?
Measure the file by its requests, not by rankings. Look in your access logs or CDN bot report for hits on /llms.txt by user agent. If no crawler or tool you care about requests it, stop spending time on it. For the effect on citations in AI answers, use the process in AI citation tracking. Do not attribute changes in rankings or AI mentions to the file without a controlled test.
What mistakes should you avoid?
- Expecting Google rankings from it. Google says Search does not use it.
- Stuffing it with keywords or sales copy. A model reading it wants facts. Google’s spam policies also cover keyword stuffing.
- Listing every URL. That is the sitemap’s job. A curated list is the point.
- Letting it go stale. Dead links in a file meant to guide tools do more harm than no file.
- Mixing it up with access control. It cannot block AI crawlers. Use robots.txt.
- Spending days on it while your pages lack clear answers, sources and structure. Work on generative engine optimization first.
The free digital marketing module of the Fe/male Switch startup school covers the SEO basics that matter more than this file. Next steps: set your crawler rules with AI crawlers and confirm every page is indexed.