Technical Requirements for AI Crawlers

What a robots.txt line controls, and what it does not

AI crawler access is governed by the ordinary Robots Exclusion Protocol — the same robots.txt file, the same syntax, standardized as RFC 9309 in September 2022 — applied to a new set of user-agent tokens. The mechanism is not new. What is new is that the AI operators have split their crawling by purpose, so one company now publishes three or four tokens that do different jobs and carry different costs when blocked.

Four limits apply to every rule you write, and three of them predate AI entirely.

• It is a crawl directive, not an index directive. A disallowed URL can still appear in results if it is linked from elsewhere. Keeping a page out requires noindex, which requires the crawler to be allowed in so it can read the tag.
• It is advisory. There is no enforcement in the protocol. It works because operators choose to comply.
• It cannot reach a crawler that does not declare itself. A blocklist of names is a blocklist of names, and names are cheap to change.
• Blocking by IP range can be worse than useless. Anthropic states directly that IP blocking "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file."

Everything below assumes you want a deliberate answer to two questions: which systems may learn from your content, and which systems may cite it. Those are separate decisions, and most published blocklists collapse them into one.

Training, retrieval and user fetches are three different decisions

Every major operator now runs at least two crawlers with different jobs, and the consequence for a publisher is asymmetric enough that getting it wrong reverses your intent.

Training crawlers gather text that may end up in a model's weights. Blocking one affects what a future model has memorized. It does not remove you from that company's live answers, because live answers are grounded on a search index rather than on the model's memory.

Retrieval crawlers build the index the assistant searches when someone asks a question. Blocking one of these is what actually removes you from AI answers.

User-initiated fetchers request a single page because a person just asked about it. Three of the four major operators state in their own documentation that robots.txt may not apply to this class at all.

The commonest expensive mistake in 2026 is blocking a training crawler and believing you have kept your content out of AI answers — or blocking a retrieval crawler and then wondering why the brand vanished from ChatGPT. Both happen because someone pasted a "block all AI bots" snippet that bundled the two categories together.

The tokens that cost you AI visibility if you block them

Block any of these and you are choosing to be absent from that product's answers. Verified against operator documentation as of 29 August 2026.

• OAI-SearchBot — OpenAI. Documented to "surface websites in search results in ChatGPT's search features" and explicitly not used for training. OpenAI states that sites opting out will not appear in ChatGPT search answers. This is the token that matters for OpenAI visibility, and it is not the one most blocklists name.
• Claude-SearchBot — Anthropic's retrieval crawler, documented as crawling the web to improve search result quality for users.
• PerplexityBot — Perplexity. "Designed to surface and link websites in search results on Perplexity"; documented as not used to crawl content for foundation models.
• meta-webindexer — Meta's retrieval crawler, documented as crawling the web to improve Meta AI search result quality.
• Googlebot — Google. There is no separate retrieval crawler for AI Overviews or AI Mode. Both are grounded on the ordinary Google Search index, fetched by Googlebot. Blocking it removes you from Google Search, which removes you from its AI features as a consequence.
• Applebot — Apple. Crawls for Spotlight, Siri and Safari search.

Two of these are worth a second look during an audit. OAI-SearchBot, Claude-SearchBot, meta-webindexer and OAI-AdsBot all postdate the blocklists still sitting in most content management system templates, so a site can be blocking a retrieval crawler it has never heard of, or allowing one it meant to block.

The tokens that only affect training

Blocking these changes what future models may have learned from your site. It does not remove you from anyone's live answers.

• GPTBot — OpenAI's training crawler, documented to "crawl content that may be used in training our generative AI foundation models." OpenAI says robots.txt changes take about 24 hours to take effect, which is the only propagation figure any operator publishes.
• ClaudeBot — Anthropic, "collecting web content that could potentially contribute to their training." Anthropic states its bots honor robots.txt and supports Crawl-delay.
• Google-Extended and Applebot-Extended — control tokens rather than crawlers. See below.
• CCBot — Common Crawl, building an open, freely redistributable corpus. This is the one decision on the list that is effectively irreversible in the other direction: data already inside released crawl archives stays there and stays redistributable, and that corpus is a long-standing input to third-party training sets. Common Crawl also warns that crawlers falsely identifying themselves as CCBot exist, a caveat that applies to every token here.
• Bytespider — ByteDance. Widely reported during 2024 as disregarding robots.txt; current third-party agent documentation says the opposite. Record it as contested. ByteDance publishes no official crawler documentation, no IP ranges and no verification method, which by itself places it in a different category from every other operator here.

Google is the special case, and the mistake is systematic

Google-Extended does not affect AI Overviews. This is the single most costly misconception in the subject and it is settled in writing, twice, by Google.

Google-Extended is not a crawler. Google's documentation: "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." It controls whether crawled content "may be used for training future generations of Gemini models" and for grounding — in Gemini apps and Vertex AI, Google's enterprise AI platform. Then the sentence that ends the argument: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

AI Overviews and AI Mode are Google Search features. Google's AI features page confirms the boundary from the other side, routing publishers to Google-Extended only to limit training and grounding in some of Google's other systems. Publishers who blocked it believing they had opted out of AI Overviews opted out of Gemini training and grounding instead, gained nothing they intended, and remained in every AI Overview they had been in. The token documentation is at Google's crawler documentation, last updated 14 July 2026.

Apple's Applebot-Extended works the same way and Apple says it more plainly than Google does: "Applebot-Extended does not crawl webpages. Webpages that disallow Applebot-Extended can still be included in search results."

The controls that do affect Google's AI features

Two mechanisms actually reach AI Overviews and AI Mode, and neither lives in robots.txt.

The snippet controls. Because eligibility for an AI Overview supporting link requires that a page be indexed and snippet-eligible, nosnippet, data-nosnippet, max-snippet and noindex all reduce or remove that eligibility. Every one of them costs you ordinary Search at the same time. They are blunt instruments and they were not designed for this.

The search generative AI control in Search Console, which took effect on 17 June 2026. It offers include, exclude or inherit-from-parent-property, covers AI Overviews, AI Mode and generative features in Discover, and takes effect within one to two days. Google states that it "only affects whether your content can appear in certain Search generative AI features; this control isn't used as a ranking or inclusion signal affecting other parts of Search," and that sites opting out will not receive traffic or impressions from the generative features. This is the lever that people reach for Google-Extended hoping to find, and it did not exist until the middle of 2026 — which is part of why the wrong token acquired its reputation.

Where robots.txt stops working

Compliance is voluntary and it has been tested in public. On 4 August 2025 Cloudflare published a study by Gabriel Corral, Vaibhav Singhal, Brian Mitchell and Reid Tatoris using brand-new, never-published domains whose robots.txt disallowed all automated access. Perplexity was then asked about those domains, and content came back. Cloudflare fingerprinted a stealth crawler using a generic Chrome user-agent string, rotating IP addresses outside Perplexity's published ranges and switching networks, at three to six million requests a day against a declared-crawler volume of twenty to twenty-five million, and de-listed Perplexity as a verified bot. Perplexity rebutted the following day, attributing the traffic to a third-party browser service and putting its own volume under 45,000 daily requests. Both accounts are on the record; neither has been independently adjudicated.

Publisher litigation has followed the same fault line. News Corp's Dow Jones and New York Post sued Perplexity in October 2024, and Nikkei and Asahi Shimbun sued in Japan in August 2025, explicitly alleging that robots.txt was ignored. Whatever those cases decide, they are evidence that the file's authority is contested rather than assumed.

The practical response has moved to the network layer, where refusal can be enforced rather than requested. Cloudflare, which handles roughly 20% of web traffic, began blocking AI crawlers by default for new domains on 1 July 2025, having already recorded more than a million customers choosing to block since it shipped the option in September 2024.

llms.txt is not a crawler directive, and nothing reads it

llms.txt is routinely filed alongside robots.txt and belongs in a different category entirely. Proposed by Jeremy Howard on 3 September 2024 and revised to a second version on 10 August 2026, it is a Markdown file at /llms.txt listing a site's important pages so that a language model can find them. It grants no permission and denies none. It is a curated index offered to a consumer that has to choose to look for it.

Nothing looks for it, and this is unusually well established. Ahrefs performed a server-log measurement — reading the actual request records a web server writes for every file it serves, rather than inferring behavior from search results — across 137,210 domains in June 2026. Of those, 38,360 published an llms.txt. 97% of those files received zero requests during May 2026. Of the requests that did arrive, 96% were bots and 4% humans; SEO audit tools were the largest single category at 21.7%; and AI retrieval bots, the only consumers that could make the file affect an answer, accounted for 1.1%. The study is at Ahrefs' llms.txt server-log study.

Google is explicit: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search," and maintaining one "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." That is a statement of behavior, not an absence of interest. Adoption meanwhile grew 8.8 times in the year to May 2026, from 4,088 files to 36,120 — a practice growing nearly ninefold a year while being read by almost nobody, and the cleanest example this field offers of an idea that spread on plausibility alone.

Auditing what you actually allow

The useful exercise is not writing a new robots.txt. It is finding out what the current one already does, because most sites are running a policy nobody chose.

Start from your server logs rather than from the file. The logs show which tokens are requesting pages, at what volume, and whether the ones you thought you blocked are still arriving — which is also how you catch an impostor, since most operators publish reverse-DNS or IP-range verification and a log match without it is only a clue. Then read the file against the purpose split above and ask, token by token, whether each block was a decision or an inheritance.

Two failure modes are worth checking for by name. A site that pasted a blocklist in 2024 is probably blocking GPTBot while leaving OAI-SearchBot untouched, or the reverse, because the retrieval tokens are newer than the lists. And a site that added Google-Extended as its AI opt-out never left Google's AI features at all. Re-check on a schedule: tokens appear and change, blocklists go stale within months, and this is the rare part of AI search where the answer is knowable, checkable and entirely under your control.

Frequently asked questions