What Is llms.txt, and Does It Help AI Search?

llms.txt is a Markdown file served at the root of a website, at /llms.txt, that lists the site's most useful pages with a one-line note on each. It exists so an AI agent can find the right page without wading through menus, scripts and cookie banners to get there. Jeremy Howard proposed it in September 2024 and published a revised v2 of the proposal in August 2026. [1] It is a community convention, not a web standard, and nothing obliges any AI product to read it.

So does it help AI search visibility? Not for rankings. Google Search ignores the file. [2] SE Ranking's study of about 300,000 domains found no link between having one and being cited by AI. [3] What llms.txt does is help the agents that fetch pages while someone waits for an answer. On the docs sites we host, those agents read one page for every 3.4 that people read.

TL;DR

• llms.txt is an index for AI agents: an H1 with the site's name, a one-line summary, and sections of links to Markdown pages.
• Google Search ignores it, and no AI search provider has said it uses the file to rank or cite pages.
• AI assistants already read docs at scale. Over ten weeks, assistants fetching pages for a live user read one docs page for every 3.4 that people read.
• They read far more than they send. For every visitor AI tools sent to a docs site, assistants read about 56 pages. Most AI visibility happens without a visit.
• AI search indexing grew tenfold in six weeks. OAI-SearchBot, which builds ChatGPT's search index, read more docs pages than Googlebot and Bingbot combined.
• Most docs sites already publish one. 87% of the docs sites Agent Score has scanned have an llms.txt, but most files have problems: 37% run past 50,000 characters, 28% leave out pages, and 80% serve Markdown that disagrees with the HTML page.
• Generate it, keep it short, keep it in sync, and track how often agents read your pages as well as how many clicks they send.

What Is llms.txt?

Where a sitemap lists every URL for a crawler, llms.txt lists only the pages worth reading, each with a note a model can use to decide whether to fetch it.

An HTML page wraps the few hundred words a reader came for in navigation, scripts and tracking, and an agent pays for every token of it. The proposal puts it plainly: context windows "are still too small for most websites in their entirety, and every wasted token costs time and money." [1] An index lets the agent pick the right page from one short file, then fetch that page as Markdown instead of HTML.

The index rarely travels alone, and the files around it are easy to confuse:

• /llms.txt is the index: titles, links and one-line notes.
• A Markdown version of each page is what the index links to. The proposal puts it at the page's own URL with .md added or swapped in, so /docs/auth becomes /docs/auth.md. [1]
• /llms-full.txt is the whole site's content in one file. It is not part of the proposal. It is a convention some docs platforms added on top, and not every platform generates it. It suits loading a small site into a context window in one go, and is impractical for a large one.

What Does an llms.txt File Look Like?

The format is ordinary Markdown in a fixed order. An H1 with the name of the site is the only required part. After it come a blockquote summary, optional paragraphs of context with no headings, and H2 sections whose bodies are lists of links, each with an optional note after a colon. [1] Here is an excerpt from the one on this site, at documentation.ai/llms.txt, with most of the links cut:

# Documentation.AI

> Create and maintain world-class documentation built for both humans and AI. The AI native documentation and knowledge platform.

This file indexes documentation.ai. Product documentation is a separate application served under /docs on this same host and publishes its own index at https://documentation.ai/docs/llms.txt, where every page is also available as Markdown by appending .md to its URL.

## Product

- [Pricing](https://documentation.ai/pricing): All four plans with per-feature comparison: free tier, seats, AI usage limits, private docs, authentication, and analytics
- [Agent Readiness Score](https://documentation.ai/agent-score): Free tool that scores how well AI agents can find, read and answer from any docs site or help center, with every check explained and a fix for each problem

## Optional

- [Get a demo](https://documentation.ai/get-a-demo): Book a walkthrough with the team
- [Contact](https://documentation.ai/contact): Support, sales, community, and documentation links

Here is what each part of that file does:

1. The blockquote is the summary a model reads first. It is often the only line read before deciding whether the site is relevant, so it should describe the product in plain terms.
2. The paragraph after it routes the agent. This site and its product docs are separate applications, so the file says where the second index lives instead of copying 120 links that would go stale. The proposal backs this up: a file covers the URLs under its path, so /docs/llms.txt covers everything in /docs/, and an agent uses the most specific file that applies. [1]
3. Each note says what is on the page. A meta description is written to win a click from a person. An llms.txt note is written so an agent can skip a page that will not answer its question.
4. ## Optional holds what can be dropped. By convention it lists secondary links "an agent can skip when a shorter context is needed", so demo, contact and legal pages go there, and product and documentation pages do not. [1]

A file can also carry instructions. Stripe's docs index skips the summary and opens with guidance for coding agents, such as checking the npm registry for the current package version instead of trusting one from training data.

llms.txt vs robots.txt vs sitemap.xml

The three files sit side by side at the root of a site, and each answers a different question for a different reader.

llms.txt cannot let an agent in or keep one out. If robots.txt blocks ChatGPT-User or OAI-SearchBot, or a bot challenge stops them at the edge, a perfect llms.txt changes nothing, so check robots.txt first.

Who Reads Your Documentation?

Before asking whether AI tools read llms.txt, we looked at who reads documentation at all. We took every page view recorded on the docs sites we host between 14 August and 22 October 2026, from browsers and from bots, and grouped each visitor by the purpose its operator documents for it. [4]

What the Data Shows

• People are a minority of readers. Humans made 11% of page views. AI training crawlers made almost three times as many, and Meta's crawler alone made 1.7 times as many.
• 3.2% came from assistants answering a live question. This is the traffic llms.txt is built for. ChatGPT-User made 59% of it, Claude-User 26%, YouBot 7% and Perplexity-User 2%.
• ChatGPT out-read Google. ChatGPT-User fetched 40% more pages than Googlebot crawled.
• AI search indexing jumped tenfold in late September. Most of the rise came in the week of 21 September and held through October. OAI-SearchBot, which builds ChatGPT's search index, made more page views than Googlebot and Bingbot combined. [4]
• Coding agents are undercounted. Under 0.1% of views named a coding tool such as Cursor. Many coding agents identify as axios or python-requests instead, and those clients made more than 50 times as many views.

Limits of This Data

• Human views come from a browser script, so readers with blockers are missing and the human share is a floor.
• Every site is hosted by us, with server-rendered pages and llms.txt on by default, so agents had an easy time reading them.
• The figures count HTML page views, not requests for llms.txt itself.

Does llms.txt Help AI Search Visibility?

It helps agents that fetch pages on demand, and there is no evidence it helps search indexes or rankings.

Google Search: No

• Google's guide to generative AI features lists "creating unnecessary AI text files (like llms.txt)" among tactics site owners can ignore, "as Google Search ignores them." [2]
• Its page on AI features says no new machine-readable files are needed to appear in them. [2]
• Chrome's Lighthouse has an experimental llms.txt audit, but it covers agents working in a browser, not search ranking. [5]

AI Search Indexes: Not Directly

• OpenAI, Anthropic and Perplexity document what their crawlers do, and none say the crawlers read llms.txt. [4]
• A page gets into ChatGPT's or Perplexity's index the same way it gets into Google's: the crawler is allowed in, the text is in the served HTML, and the URL is linked from somewhere.

Agents Answering a Question: Yes

• This is the use the proposal was written for. An assistant asked how to rotate an API key reads the index, picks one link and fetches that page as Markdown, instead of starting at the homepage and guessing.
• The audience is narrow. In Ahrefs' study of 137,210 domains, 97% of llms.txt files got no requests at all in May 2026, and among AI tools only GPTBot fetched the file more often than Claude Code. [6]

Clicks From AI: Barely

Over the same ten weeks, AI assistants read far more of our docs than they sent readers to.

• ChatGPT sent 86% of the visitors that came from AI tools.
• AI tools sent 7.6% of the visits that came from search engines and AI tools combined. Search engines sent the rest.
• Some AI clicks arrive without a referrer, so AI sends more visitors than this shows, though nowhere near enough to close a 56-to-1 gap.
• An assistant reads the page, answers the question, and the reader stays in the chat.

With assistants reading that much more than they send, what matters for documentation is whether the answers they give from your pages are correct. llms.txt helps by steering an agent already on your site to the right page, which also explains why SE Ranking found it does not predict citations. [3]

How Many Docs Sites Have llms.txt, and What Do They Get Wrong?

Most docs sites already publish one. Agent Score runs the open-source AFDocs checks against any documentation site, and 87% of the sites scanned so far have an llms.txt. Nearly every site on a hosted docs platform had one, and almost all of the sites without a confirmed file were built and hosted by the team itself.

The sample is self-selected: these are sites someone chose to scan, weighted toward hosted platforms. Read the percentages as a picture of docs teams who already care about agents, not of the whole web, where Ahrefs put adoption at 28% at most. [6] Even in this group, most files have something wrong with them.

Most of these problems come from a file that has stopped matching the site:

• Written once and never updated. A file written by hand at launch misses every page added since.
• Built from an old export. It sends agents to Markdown copies that no longer match the live page.
• Every page listed. On a site with thousands of pages, the file grows past what an agent will read. On 15% of the sites scanned, it runs past 100,000 characters.

None of this shows up in a browser. The first sign is usually an assistant giving a wrong answer from a page that is correct on your site.

How to Create an llms.txt File

Use the file your platform generates, and write one by hand only if it generates nothing.

• Hosted docs platforms mostly generate it automatically or behind a setting. Documentation.AI rebuilds it on every publish, grouped by navigation section, with each link pointing to the page's Markdown.
• WordPress sites get one from Yoast SEO's free version once you switch it on.
• Static site generators usually need a community plugin.
• Custom sites should generate it from the same source as the navigation or sitemap, so a new page reaches the index the day it ships.

Whether it is generated or written by hand, the file should meet these rules:

1. Pick the pages. Include pages that answer questions: guides, API reference, setup, troubleshooting, pricing. Put demo, contact and legal pages under ## Optional, and leave out tag archives and search pages.
2. Write the header. One # Name, one > summary that says what the product does in a sentence, and an optional paragraph that routes agents to anything the file does not cover.
3. Group by task. Use H2 sections that match how people look for things, such as Getting started, Guides, API reference and Troubleshooting, not your org chart.
4. Link to Markdown. Point each entry at the page's .md URL where one exists, with an absolute URL.
5. Write notes for an agent. Say what is on the page in under 25 words, so the agent can skip it when it is the wrong one.
6. Keep it under 50,000 characters. On a large site, publish one llms.txt per section, such as /docs/api/llms.txt, and link to them from the root file instead of listing every page.
7. Point to it from every page. Add one short line near the top of each page, such as > For AI agents: a documentation index is available at /llms.txt. The v2 proposal also suggests an HTTP Link header that points to it. [1]
8. Serve it plainly. Return 200 with text/plain or text/markdown, with no login, no redirect chain and no bot challenge in front of it.

How to Check That Your llms.txt Works

Run these checks after setting it up, and again after any large docs change.

1. Fetch it the way an agent does. curl -sI https://yoursite.com/llms.txt should return 200 and a text content type, and curl -s https://yoursite.com/llms.txt | wc -c should come in under 50,000.
2. Open three links at random. Each should return Markdown with the same content as the HTML page, not a 404 or an HTML page.
3. Read your server logs. grep "llms.txt" access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn lists who fetched the file, by user agent, in the common combined log format. Run the same command for your .md URLs to see whether agents follow the links.
4. Run a scan. The free Agent Readiness Score runs every check in the chart above against your site, names the failing pages, and gives the fix for each one.

The guide to AI documentation for agents and LLMs covers the work around the file, from server rendering to Markdown parity, and What Is WebMCP? covers the layer that may come after it.

FAQs

1. What is llms.txt?

llms.txt is a Markdown file at the root of a website, such as example.com/llms.txt, that links to the site's most useful pages with a short note on each. AI agents read it to pick the right page without crawling HTML, then fetch a clean Markdown copy of that page.

2. Does llms.txt help SEO?

Not for search rankings. It does not change how Google crawls or ranks pages, and it does not replace robots.txt or a sitemap. Where it helps is AI answers: agents that fetch your pages find the right one faster, so what they say about your product is more accurate.

3. Does Google use llms.txt?

No. Google says Search ignores llms.txt, so it neither helps nor harms rankings. [2] The only Google tool that checks for it is Lighthouse, in an experimental audit for AI agents working in a browser. [5]

4. Does ChatGPT read llms.txt?

OpenAI has not said that ChatGPT uses llms.txt, and its crawler documentation does not mention the file. [4] Its GPTBot crawler does fetch llms.txt files: in Ahrefs' study it fetched more of them than any other named AI tool. [6] When ChatGPT browses a site to answer a question, it can follow the file's links like any other page.

5. What is the difference between llms.txt and robots.txt?

robots.txt tells crawlers what they may fetch, and crawlers that follow the convention obey it. llms.txt tells AI agents which pages are worth reading and where the Markdown copy is. It cannot block or allow anything. A site needs both, and a robots.txt that blocks AI crawlers makes llms.txt pointless.

6. What is the difference between llms.txt and llms-full.txt?

llms.txt is an index of links with short notes. llms-full.txt puts the full text of a site in one file, which some docs platforms generate although the proposal does not define it. Agents use the index to choose a page, and the full file only works for sites small enough to fit in a context window. Of the docs sites Agent Score has scanned, 48% publish one.

7. How do I create an llms.txt file?

Check whether your platform generates one first. Most hosted docs platforms do, and WordPress sites can switch one on in Yoast SEO. To write one by hand, start with # Site name, add a one-line > summary, then list your key pages under ## sections as Markdown links with a short note after each. Save it as plain text at https://yoursite.com/llms.txt.

8. Is llms.txt an official standard?

No. It is a community proposal that Jeremy Howard published in September 2024 and revised as v2 in August 2026. [1] No standards body such as the IETF or W3C has adopted it, and no AI company is obliged to read it.

Sources

1. Howard, Jeremy. "The /llms.txt file, v2." llmstxt.org, 3 September 2024, revised August 2026. https://llmstxt.org/
2. Google. "Google Search Central Docs." Google for Developers, accessed 23 October 2026. https://developers.google.com/search/docs
3. SE Ranking. "LLMs.txt: Why Brands Rely On It and Why It Doesn't Work." SE Ranking Blog, 15 October 2026. https://seranking.com/blog/llms-txt/
4. OpenAI. "Overview of OpenAI Crawlers." OpenAI Developers, accessed 22 October 2026. https://developers.openai.com/api/docs/bots
5. Chrome for Developers. "llms.txt." Lighthouse documentation, 5 May 2026. https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt
6. Linehan, Louise. "We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read." Ahrefs Blog, 15 June 2026. https://ahrefs.com/blog/llmstxt-study/