AI Crawler Directives and llms.txt

Two files, and only one of them is a permission system

AI crawler directives are entries in a site's robots.txt naming the user-agent tokens that AI companies use, and allowing or disallowing them. The mechanism is the ordinary Robots Exclusion Protocol, standardized as RFC 9309 in September 2022 — the same file and the same syntax that has governed every web crawler, the automated program that fetches pages to build an index, since the 1990s.

The genuinely new idea is that AI operators have split their crawling into purposes: one token for gathering training data, another for building the index that powers an assistant's answers, a third for fetching a single page because a user just asked about it. Block the wrong one and you lose visibility you wanted. Block only the obvious one and you leave open the door you meant to close.

llms.txt is a different and much weaker proposal: a Markdown file at /llms.txt listing a site's most important pages so a language model can find them, proposed by Jeremy Howard on 3 September 2024 and revised to a second version on 10 August 2026. It is not a permission file and it has no relationship to robots.txt. Its intended function is curation, not control. As of August 2026 no major AI company has said it reads llms.txt for web content, and Google has said in writing that it ignores it.

Training, retrieval, and the fetch a user just asked for

Every major operator now runs at least two crawlers with different jobs, and the consequences for a publisher are asymmetric.

• Training crawlers — GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, Bytespider — gather text that may end up in a model's weights. Blocking them affects what a future model has absorbed. It does not remove you from that company's live answers, because live answers are grounded on a search index rather than on memory.
• Retrieval crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer, and for Google simply Googlebot — build the index the assistant searches when someone asks a question. Blocking these is what actually removes you from AI answers.
• User-initiated fetchers — ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher — request one page because a person just asked about it. Three of those four operators state in their own documentation that robots.txt may not apply to this class.

The commonest expensive mistake in 2026 is blocking a training crawler and believing you have protected content from AI answers, or blocking a retrieval crawler and wondering why the brand vanished from ChatGPT. OpenAI documents GPTBot as crawling content that may be used in training its foundation models, and OAI-SearchBot as surfacing websites in ChatGPT's search features — and states that sites opting out of the second will not appear in ChatGPT search answers. Two different decisions, two different costs. Most block-all-AI-bots snippets in circulation make both at once, silently, on behalf of someone who meant to make one.

The user-initiated fetchers also differ from each other, and treating them as interchangeable is wrong on one case. OpenAI hedges, saying that because these actions are initiated by a user, robots.txt rules may not apply. Perplexity says its user agent generally ignores robots.txt rules. Meta says meta-externalfetcher may bypass robots.txt because it performs fetches requested by the user. Anthropic states its bots respect do-not-crawl signals by honoring industry standard directives in robots.txt and publishes no user-initiated exception at all, which makes Anthropic and Google the only two operators here without a carve-out.

Google-Extended is not a crawler, and it does not touch AI Overviews

This is the single most costly misconception in the subject, and it is worth correcting flatly.

Google has no retrieval crawler for AI Overviews or AI Mode. Those features are grounded on the ordinary Google Search index, fetched by Googlebot. There is no separate token, and there never was.

Google-Extended is not a crawler at all. Google's own crawler documentation, stamped 14 July 2026, says it does not have a separate HTTP request user agent string, that crawling is done with existing Google user agent strings, and that the robots.txt token is used in a control capacity. What it controls is whether content Google crawls may be used for training future generations of Gemini models and for grounding. And then the sentence that settles the argument:

Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.

AI Overviews and AI Mode are Google Search features. Google's AI features page confirms the boundary from the other side, directing publishers to Google-Extended only to limit AI training and grounding in some of Google's other systems — the Gemini apps and Vertex AI, Google's cloud platform for building AI applications, which offers grounding against Google Search as a service. Blocking Google-Extended removes you from Gemini app grounding and from Gemini training data. It does not remove you from a single AI Overview.

The claim spread for two reasons worth naming. The token's name reads like Google's AI thing, and Google announced it in September 2023 with considerably less clarity than it uses now. Publishers who blocked it believing they had opted out of AI Overviews opted out of something else entirely and gained nothing they intended.

Apple's Applebot-Extended works the same way and Apple says so more plainly: it does not crawl webpages, and webpages that disallow it can still be included in search results. A use-of-data token, not a crawler.

The only levers that touch AI Overviews are the ordinary snippet controls — nosnippet, data-nosnippet, max-snippet and noindex — and every one of them costs you ordinary Search as well. Google documents no way to appear in Search but not in AI features.

llms.txt: what the server logs say

The proposal describes itself as a way to provide information to help agents use a website. It is Markdown: a heading with the site name, an optional summary, then lists of annotated links to key pages. It says nothing about permissions. It is a curated index, offered voluntarily, for a consumer that has to choose to look for it.

Nobody looks for it. Ahrefs — a search analytics company with its own web crawler and index — published a server-log study on 15 June 2026, authored by Louise Linehan and Xibeijia Guan. A server-log measurement reads the web server's own record of every request made for a file, so it counts what actually happened rather than what was claimed, which makes it the strongest evidence available on a question like this. They took 137,210 domains, identified which served an llms.txt returning HTTP 200 with real Markdown in it, and read the request logs for those files across May 2026.

• 38,360 of the 137,210 domains, or 28%, published an llms.txt.
• 97% of those files received zero requests during the month.
• Of the requests that did arrive, 96% came from bots and 4% from humans.
• SEO audit tools were the largest single requester category at 21.7%, with a further 12% coming from tools studying llms.txt itself.
• AI retrieval bots — the ones that would have to read it for it to affect any answer — accounted for 1.1% of requests.
• AI training crawlers totalled 5.3%: GPTBot at 4.51%, ClaudeBot at 0.80%, DeepseekBot at 0.02%.

The 28% publication rate is high, and the sampling caveat belongs with it — Ahrefs Web Analytics users skew toward marketing-instrumented sites. The 97% figure is the one that matters, and it is a direct measurement rather than an inference.

Adoption is nevertheless growing quickly. Originality.ai's tracker, scanning more than three million sites, recorded llms.txt going from 4,088 instances in June 2025 to 36,120 in May 2026 — a factor of 8.8 in twelve months — with llms-full.txt at 2,463 and ai.txt at 397. A file that grows 8.8 times a year and is read by nobody is the cleanest example in this whole field of an optimization that spread on plausibility alone. Growth here measures how many people read the same advice, not whether the advice was correct.

Google's position is the most explicit statement any operator has made. In the mythbusting section of its May 2026 generative-AI guide it says publishers do not need to create new machine-readable files, AI text files, markup or Markdown to appear in Google Search, and that maintaining such files will neither harm nor help a site's visibility or rankings as Google Search ignores them. That is a statement of behavior, not an absence of interest. John Mueller had already posted on 17 June 2025 that no AI system currently uses llms.txt, and Gary Illyes was reported from a Search Central event in July 2025 as saying Google does not support it and does not plan to — though that account reaches us through an attendee rather than a transcript.

One last argument deserves dismantling, because it is the one advocates reach for. OpenAI, Anthropic and Google all publish llms.txt files — for their own developer documentation, so coding assistants can work through their API docs. That is a company acting as a publisher. It is not a statement that its crawlers consume the format when reading the rest of the web, and no operator has made that statement.

robots.txt is a request, not a fence

Everything above assumes the file works as intended. Several things limit it, and most of them predate AI entirely.

It is a crawl directive, not an index directive. A disallowed URL can still appear in results if it is linked from elsewhere. Keeping a page out requires noindex, which in turn requires the crawler to be allowed in so it can read the tag. This ordinary error is being re-imported wholesale into AI-crawler advice.

It is advisory, and it relies on the operator choosing to obey. There is no enforcement in the protocol, and it cannot reach a crawler that does not declare itself. Cloudflare documented exactly that failure mode on 4 August 2025: it registered brand-new domains, never published, with a robots.txt disallowing all automated access, then asked Perplexity about them and got content back. It fingerprinted a stealth crawler using a generic Chrome user-agent string, rotating IPs outside Perplexity's published ranges and switching networks, running 3 to 6 million requests a day against a declared-crawler volume of 20 to 25 million. Cloudflare's conclusion was that Perplexity was repeatedly modifying its user agent and changing source networks to hide its crawling activity, as well as ignoring or sometimes failing even to fetch robots.txt files. Cloudflare de-listed Perplexity as a verified bot.

Impersonation cuts the same way from the other end. Common Crawl warns that crawlers falsely identifying themselves as CCBot exist, and publishes reverse-DNS and IP-range verification — a caveat that applies to every token in every blocklist and that almost no blocklist advice mentions. Bytespider publishes no verification method at all, so any client can claim its identity and a log match is only a clue. Blocking by IP address is worse than useless against operators that fetch robots.txt from those same ranges: Anthropic states directly that IP blocking may not work correctly or persistently guarantee an opt-out, because it impedes its ability to read your robots.txt file in the first place.

The infrastructure answer has accordingly moved to the network layer. Cloudflare, which handles roughly 20% of global web traffic, began blocking AI crawlers by default for new domains on 1 July 2025, having reported more than a million customers already choosing to block since the option shipped. That change did more to determine what AI systems can read than any robots.txt snippet published that year.

What to check, and how often

The practical work here is small, and most of it is auditing rather than adding.

• Decide training and retrieval separately. Blocking training while remaining citable is a coherent and common position. Blocking both because a blog post bundled them is not a decision anyone made deliberately.
• Re-read your robots.txt against current tokens. OAI-SearchBot, Claude-SearchBot, meta-webindexer and OAI-AdsBot all postdate the blocklists still shipping in most CMS templates. A list audited once goes stale within months.
• Do not block Google-Extended to escape AI Overviews. It will not work, and you will lose Gemini grounding you may have wanted.
• Treat CCBot as the irreversible one. Common Crawl's corpus is a long-standing input to third-party training sets, and data already in released archives stays there and stays redistributable.
• Publish llms.txt if you like, and do not report it as a result. Google says it neither harms nor helps, and 97% of these files are never requested.

The underlying lesson generalizes past this subject. A blocklist of names is a blocklist of names, and names are cheap to change. Robots.txt directives are a statement of intent that well-behaved operators honor and that nothing enforces, and the operators themselves have written down where they stop honoring it. Anyone who needs a guarantee rather than a request is working at the wrong layer.

Frequently asked questions