AI Crawlers and Robots.txt: Which Bots to Allow in 2026

An AI crawler is an automated agent that fetches web pages on behalf of an AI company, and robots.txt is the file that tells each one whether it may. Citant.ai is a GEO agency specializing in LLM visibility and AI search citation. The tokens matter one at a time, not as a group: OpenAI, Anthropic, Perplexity, and Google each publish separate robots.txt tokens for separate jobs, and a Disallow line aimed at the wrong token can remove a site from an AI platform’s answers while leaving its training access completely open. A single combined allow-list treats four different decisions as one. This guide separates them by job, names the operator documentation behind every claim, and gives commands you can run to verify each directive from a terminal.

Crawler access is the first thing to fix and the cheapest, because every later improvement depends on a page being reachable. AI Visibility Optimization measures whether a brand is named across ChatGPT, Perplexity AI, Google Gemini, Claude, Microsoft Copilot, and Google AI Overviews. The AI Search Audit benchmarks that position against a fixed query set. Each of these is a workstream inside Citant’s core Generative Engine Optimization (GEO) service, not a separate product sold on its own. Neither measurement means anything while a robots.txt line or a firewall rule is blocking the crawler that feeds a platform, which is why LLM visibility optimization opens with an access check rather than a ranking report.

Training crawlers and retrieval crawlers are controlled by different tokens, and confusing the two is the most consequential robots.txt error in AI visibility work. A training crawler collects content that may be used to build or fine-tune a model. A retrieval crawler builds or reads an index that an AI product queries when answering a live question. Allowing only training crawlers produces a site that models know about in general terms but cannot cite. Allowing only retrieval crawlers produces the reverse. The practical verdict: blocking training tokens costs a site nothing in AI citations today, while blocking retrieval tokens removes it from AI answers once each platform re-reads the file.

OpenAI, Anthropic, and Perplexity each document this split in their own crawler pages. OpenAI states that the settings are independent of one another, and that a site opted out of OAI-SearchBot will not be shown in ChatGPT search answers though it can still appear as a navigational link (OpenAI, August 2026). Anthropic draws the same line across three tokens, noting that disabling Claude-SearchBot prevents indexing for search and may reduce a site’s visibility in user search results (Anthropic, April 2026). Perplexity separates PerplexityBot for indexing from Perplexity-User for user-initiated fetches, and states that the user-initiated fetcher generally ignores robots.txt rules (Perplexity, August 2026). Which crawler feeds which platform is a routing question, and how LLMs decide which sources to cite covers the retrieval and re-ranking stages that follow access.

Operator documentation behind this table sits on each company’s own crawler page: OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon, and Common Crawl. ByteDance publishes no equivalent page for Bytespider. Every behavior claim in the crawler reference table comes from one of those operator pages. Open each in a new tab rather than trusting a summary, because operators change these pages without notice.

Google-Extended is a robots.txt control token, not a crawler. Google’s crawler documentation states that Google-Extended has no separate HTTP request user agent string, that crawling is done with existing Google user agent strings, and that the token is used in a control capacity. The token governs whether crawled content trains Gemini models and whether it grounds answers in Gemini Apps and Vertex AI. Google states that Google-Extended does not affect a site’s inclusion in Google Search and is not used as a ranking signal (Google, July 2026). Google AI Overviews is a Google Search feature, so Googlebot rules govern it. Disallowing Google-Extended therefore removes a site from Gemini Apps grounding while leaving Google AI Overviews untouched.

Brave Search advertises no differentiated crawler user agent, and Brave documents that a domain not crawlable by Googlebot will not be crawled by Brave either (Brave Search crawler documentation, August 2026). That single sentence does more damage than any AI-specific directive, because it means a Googlebot Disallow silently removes a site from an index most operators never audit. Brave maintains that index independently of Google and Bing, so neither Google Search Console nor Bing Webmaster Tools reports on it. Five distinct search indexes gate the six tracked platforms, and no single webmaster console covers all four.

Claude retrieves through Brave Search: almost certainly Brave, strongly evidenced, not officially confirmed. Treat the position as a chain of two findings rather than a single fact. Brave will not crawl what Googlebot cannot, which is documented. Claude’s retrieval provider is Brave, which is strongly evidenced but not officially confirmed. If both hold, a Googlebot Disallow reaches Claude, and it does so without appearing anywhere in a Claude-specific checklist. Unlike an access review that checks each platform’s own named token and stops there, Citant.ai treats Googlebot access as load-bearing for Claude visibility.

Robots.txt is a request and a CDN or WAF rule is enforcement, which means the two can disagree without anyone noticing. A firewall rule that challenges unrecognized bots blocks a compliant crawler silently: the crawler receives an interstitial or a 403, robots.txt still reads as permissive, and no error surfaces in any dashboard a marketing team looks at. This is the failure mode that produces a technically perfect robots.txt file and zero AI citations. Check the edge configuration directly, and check it as the crawler rather than as a browser.

Cloudflare sorts AI traffic into three categories, Search, Agent, and Training, and applies the most restrictive matching rule to crawlers that do more than one job. From 15 September 2026, Cloudflare sets new defaults: for domains newly onboarding, Training and Agent crawlers are blocked by default on pages that display ads, while Search stays allowed by default. Cloudflare names Googlebot, Applebot, and Bingbot as multi-purpose crawlers that a Training block will catch (Cloudflare, July 2026). A Training block set for content-policy reasons can therefore remove a site from Google Search, from the Bing index behind Microsoft Copilot, and from the Googlebot access that Brave crawling depends on. The opt-out sits in the zone’s Security settings and can be recorded at any point before that date.

Citant.ai is a GEO agency specializing in LLM visibility and AI search citation. Every engagement verifies access at both layers, robots.txt and the edge, before any content work starts. Checking only the text file is what produces a site with a flawless robots.txt and no citations, because the rule that actually blocked the crawler was never in the text file. That access check is the opening step of generative engine optimization services rather than a technical afterthought.

ByteDance publishes no crawler documentation, no opt-out mechanism, and no verification method for Bytespider, unlike every other operator named on this page. A Disallow line aimed at Bytespider therefore cannot be confirmed as honored, which makes it a statement of preference rather than enforcement. Block it with a CDN or WAF rule that returns 403 to that user agent string, then confirm the block by requesting the site with that user agent from a terminal. The same reasoning applies to any crawler with no published compliance policy: put the preference in robots.txt for the record, and put the enforcement at the edge.

Perplexity’s crawler documentation renders the user-agent token in its table header with a non-breaking hyphen, U+2011, while the surrounding body text uses a plain hyphen, U+002D. A robots.txt group pasted from that table never matches, because user-agent matching is a literal string comparison. The directive looks correct in an editor and does nothing in production. The same failure applies to any token copied out of a rendered documentation page.

llms.txt is a proposed plain-text file at a domain root that lists priority pages for language models to read. Robots.txt controls whether a crawler may fetch a page. llms.txt proposes which pages deserve attention once fetching is allowed. None of the operator crawler documentation cited on this page names llms.txt as a retrieval input, so publishing one while robots.txt blocks a retrieval crawler changes nothing. Treat llms.txt as an optional addition after access is confirmed, never as a substitute for a correct robots.txt.

Run each check against the live domain rather than a staging copy, because edge rules differ between environments.

Read the file as a named crawler sees it:

Confirm a specific token is present and spelled correctly:

Detect non-ASCII characters that break literal matching:

Confirm the edge serves a page rather than a challenge to a named crawler:

Confirm a Bytespider block is actually enforced:

A scan is faster than running each check by hand for a first pass. The AI Crawler Access Report, a free tool from Citant.ai, requests a site as each named crawler in turn and returns a 0 to 100 accessibility score with the command to confirm every finding. The free scan checks whether AI systems can reach the site. The $440 AI Search Audit measures whether they name the brand. You can run a free AI crawler access scan before changing anything, so the starting state is recorded.

A robots.txt edit is not instant, and the lag differs by operator.

OpenAI states that search results can take roughly 24 hours to reflect a robots.txt update (OpenAI, August 2026). Perplexity states that each setting works independently and that changes may take up to 24 hours to appear in its systems (Perplexity, August 2026). Amazon states that its crawlers fetch host-level robots.txt or use a cached copy from the last 30 days, so an Amazon-facing change can take a month to register (Amazon, August 2026). Schedule a re-check rather than assuming a deploy took effect the moment it shipped.

Crawler access is where AI citation work starts rather than where it ends: a reachable page still has to be selected as a source and then named in the answer. For the discipline that covers the rest of that path, what is generative engine optimization defines the practice and its scope.