AI Crawler Access Report: Check Whether AI Systems Can Read Your Site
The AI Crawler Access Report is a free tool from Citant.ai that tests whether AI systems can technically reach and read a website. Access is the first failure point in that work: a page no crawler can fetch cannot be retrieved, re-ranked, or named in an AI answer, no matter how well it is written.
- Free, no signup
- 15 named AI crawlers tested
- First results in seconds
Run the free scan
No signup required to see the score. The scan returns a 0 to 100 AI accessibility score and a command you can run yourself to confirm every finding.
What the AI Crawler Access Report Checks
The Report requests pages while identifying as fifteen named crawlers in turn, then reports each one as reachable or blocked.
Naming matters here: a robots.txt rule that blocks GPTBot does not block OAI-SearchBot, and a rule that blocks a training crawler does not block the crawler that fetches pages during a live conversation. Site owners routinely block one and assume they have blocked all three, or allow one and assume they have allowed all three.
Citant.ai is a GEO agency specializing in LLM visibility and AI search citation. A blocked live-answer crawler is the most common technical finding in that work, and the fix is usually a single line in a file nobody has opened in two years.
| Crawler | Operator | Category | Sub-score it feeds |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Live answer | Live Answer Visibility |
| ChatGPT-User | OpenAI | Live answer | Live Answer Visibility |
| PerplexityBot | Perplexity | Live answer | Live Answer Visibility |
| Perplexity-User | Perplexity | Live answer | Live Answer Visibility |
| Claude-SearchBot | Anthropic | Live answer | Live Answer Visibility |
| Claude-User | Anthropic | Live answer | Live Answer Visibility |
| DuckAssistBot | DuckDuckGo | Live answer | Live Answer Visibility |
| GPTBot | OpenAI | Training | Training Data Presence |
| ClaudeBot | Anthropic | Training | Training Data Presence |
| CCBot | Common Crawl | Training | Training Data Presence |
| meta-externalagent | Meta | Training | Training Data Presence |
| Amazonbot | Amazon | Training | Training Data Presence |
| Bytespider | ByteDance | Training | Training Data Presence |
| Googlebot | Search index | Search Index Foundation | |
| bingbot | Microsoft | Search index | Search Index Foundation |
The three categories fail differently. A blocked live-answer crawler removes a site from AI answers now. A blocked training crawler removes it from the data that shapes what an AI system says about a brand when it answers without retrieving anything. A blocked search-index crawler removes it from the indexes that the live-answer crawlers draw on.
Three files fetched directly
- robots.txt reported as present or absent, with its parsed contents where present.
- The XML sitemap reported as present or absent, with its parsed contents where present.
- llms.txt reported as present or absent, with its parsed contents where present.
How the AI Accessibility Score Works
The AI Crawler Access Report returns a single 0 to 100 AI accessibility score built from three weighted sub-scores. Each sub-score maps to a different way AI systems encounter a website, and a site can pass one while failing another.
Live Answer Visibility
Measures whether the seven live-answer crawlers can reach the site, whether the page renders without JavaScript, and whether it is indexable. Crawler access carries 50 percent of the sub-score, renderability 35 percent, and indexability 15 percent. Renderability weighs heavily because a crawler that is allowed in and receives an empty shell has been blocked in practice.
Search Index Foundation
Measures whether the search indexes that feed live retrieval can reach the site. Googlebot and bingbot access carries 30 percent, indexability including sitemap presence carries 40 percent, and infrastructure carries 30 percent. Four separate indexes gate the six tracked platforms, so index-level access is not a single check.
Training Data Presence
Measures whether the six training crawlers can reach the site, whether the domain appears in Common Crawl, and whether the page renders without JavaScript. When the Common Crawl API is unreachable, its weight is redistributed across the remaining inputs rather than scored as a failure.
The composite score weights Live Answer Visibility at 45 percent, Search Index Foundation at 30 percent, and Training Data Presence at 25 percent.
Retrieval and naming are two different stages of the same process, and how LLMs decide which sources to cite covers what happens after a page is successfully fetched.
What Each Score Band Means
The AI accessibility score is banded so that a number resolves to an action. Bands are absolute, not relative to other sites.
| Score | Band | What it means | What to do next |
|---|---|---|---|
| 90 to 100 | Fully open | Every tested crawler reaches the site, the page renders server-side, and the supporting files are present and valid | Re-run quarterly and after any robots.txt, hosting, or CDN change |
| 75 to 89 | AI-ready | Live-answer access is clean, with a gap in training access, sitemap coverage, or infrastructure | Fix the named gap, then re-scan |
| 50 to 74 | Accessible with leaks | The site is reachable, but at least one category has a real blocker inside it | Work down the findings by category, starting with live answer |
| 26 to 49 | Partially blocked | Multiple crawlers are blocked, or one CRITICAL finding is capping the score | Fix the CRITICAL finding first, then re-scan before touching anything else |
| 0 to 25 | Invisible | AI systems cannot meaningfully reach or read the site | Treat as a technical incident. Content work is wasted until this is fixed |
Any single CRITICAL finding caps the composite score at 49, regardless of how every other check performed. A noindex directive on the homepage caps the composite score at 25.
Both caps are deliberate. A site with one CRITICAL blocker is not partly accessible to AI systems, and a homepage carrying noindex is not accessible at all, so an averaged score would report a healthier site than the one that exists.
How to Verify Every Finding Yourself
Every finding in the AI Crawler Access Report ships with the command that produces it, so no result has to be taken on trust. Run these against any domain to reproduce the scan by hand.
Run curl.exe, not curl. In PowerShell, curl is an alias for Invoke-WebRequest, which takes different arguments and returns a different object, so every command below will either error or produce output that does not match the Report. Substitute curl.exe for curl in all eight steps.
-
Fetch the robots.txt file and read it in full
curl -s https://example.com/robots.txt -
Search the output for each crawler token
Check whether a specific crawler is disallowed by searching the output for its user agent token:
OAI-SearchBot,ChatGPT-User,PerplexityBot,Perplexity-User,Claude-SearchBot,Claude-User,DuckAssistBot,GPTBot,ClaudeBot,CCBot,meta-externalagent,Amazonbot,Bytespider,Googlebot,bingbotorGoogle-Extended. -
Request the homepage as one of those crawlers
Read the status code the server returns.
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://example.com/ -
Repeat step 3 for each crawler
A 200 means the server served the page. A 403 or 404 means the server refused a request it served to a normal browser.
-
Confirm the XML sitemap resolves and carries current lastmod values
curl -s https://example.com/sitemap.xml | head -40 -
Confirm whether an llms.txt file exists at the domain root
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/llms.txt -
Confirm the page body is present in the served HTML
Copy a sentence that is visible on the rendered page, paste it in place of
PASTE_A_SENTENCE_FROM_YOUR_PAGE_HERE, keep the quotation marks, and run:curl -s https://example.com/ | grep -c "PASTE_A_SENTENCE_FROM_YOUR_PAGE_HERE"A result of 1 or higher means the text is in the served HTML. A result of 0 means either the text is added by JavaScript after load, or the sentence was not copied exactly, so check the copy before concluding the site is JavaScript-only.
-
Check server logs for real crawler visits
Filter access logs for the same user agent tokens. A crawler that is allowed but never appears in logs is a different problem from one that is blocked.
Firewalls, bot management rules, and CDN configurations block by user agent above the robots.txt layer, which is why the Report requests real pages as each crawler rather than only reading the robots.txt file.
What the Free Scan Does Not Do
The free scan checks whether AI systems can reach the site. The $440 AI Search Audit measures whether they name the brand.
That boundary has three practical consequences.
First, the Report does not measure Share of Model. It does not query ChatGPT, Perplexity, Google Gemini, Claude, Microsoft Copilot, or Google AI Overviews. A site can score 100 for access and still never be named in an answer, which is a brand definition problem rather than an access problem. Citant.ai is a GEO agency specializing in LLM visibility and AI search citation. The paid work begins where the free scan stops, and brands that need the citation side rather than the access side are served by generative engine optimization services and by LLM visibility optimization.
Second, the Report measures access, not index presence. Claude’s live web search retrieves through Brave Search, almost certainly Brave, strongly evidenced but not officially confirmed. Brave operates no differentiated crawler user agent and will not crawl what Googlebot cannot reach, so Googlebot reachability is the real gate on that path, and the Report tests Googlebot directly inside Search Index Foundation. What the Report cannot tell you is whether the site is actually present in Brave Search results today, which is an indexing question rather than an access question.
Third, the Report is a point-in-time scan. It reports what a server returned at the moment of the request. Hosting changes, firewall rule updates, and CDN configuration changes all move the result, which is why the recommended interval is quarterly and immediately after any infrastructure change.
Who the AI Crawler Access Report Is For
The AI Crawler Access Report is built for marketing leaders, CMOs, and growth directors at B2B SaaS companies who need a technical answer without opening a ticket with engineering.
It returns a result a non-engineer can read and a command an engineer can verify, which makes it usable as evidence in the conversation between the two.
Use the Report when
- You need to know whether AI systems can technically reach and read your site.
- You want a result a non-engineer can read, backed by a command an engineer can verify.
- You have just changed robots.txt, hosting, CDN configuration, or firewall rules.
- You want the access layer cleared before commissioning any content work.
It is not the right tool when
- You need to measure whether AI systems mention a brand. That is what the AI Search Audit does.
- You want a rank tracker. The Report reports no positions and no citation counts.
- You need a full-site crawl audit. The Report samples a homepage plus up to five additional pages, so it will not surface a blocked crawler on page 400 of a large site.
Citant.ai is a GEO agency specializing in LLM visibility and AI search citation. Sites past that sampling limit are handled inside a paid engagement, with scope and cost set out in full on GEO service pricing.
How the Report Compares to Checking by Hand
Every check the AI Crawler Access Report runs can be run manually. The difference is coverage and time, not access to anything private.
Citant.ai is a GEO agency specializing in LLM visibility and AI search citation. The Report was built to compress a checklist its own specialists ran by hand, and crawler access is only the first layer of that discipline, which what generative engine optimization is sets out in full.
| Method | Crawlers covered | Detects firewall and CDN blocks | Cost | Output |
|---|---|---|---|---|
| AI Crawler Access Report | Fifteen, across live answer, training, and search index categories | Yes, requests real pages as each crawler | Free, no signup | 0 to 100 score, three sub-scores, a command per finding |
| Reading robots.txt by hand | Any, if the reader knows the tokens | No, robots.txt is advisory and sits below firewall rules | Free | A text file the reader interprets |
| curl with a spoofed user agent | One per command | Yes | Free | An HTTP status code per request |
| Google Search Console and Bing Webmaster Tools | Googlebot and bingbot only, no AI-specific crawler is covered | Partially, for those two crawlers | Free, requires verified ownership | Index coverage for two search crawlers |
| Server log analysis | Every crawler that actually visited | Shows outcomes, not the rule that caused them | Free, requires log access | Historical visit records |
Unlike a robots.txt read, which reports only what a file says, the AI Crawler Access Report reports what a server actually did when a named crawler asked for a page. Results from paid engagements that began with an access problem are recorded in GEO results and client case studies, and the other free diagnostics sit on the free GEO diagnostic tools hub.
Frequently Asked Questions
How do I check if AI can read my website?
Run the AI Crawler Access Report, a free tool from Citant.ai, at check.citant.ai. It requests pages while identifying as fifteen named crawlers, fetches robots.txt, sitemap, and llms.txt, and returns a 0 to 100 AI accessibility score with a command to confirm each finding.
How do I know if ChatGPT can crawl my site?
ChatGPT reaches sites through three separate OpenAI crawlers: GPTBot for training, OAI-SearchBot for search indexing, and ChatGPT-User for fetching a page during a conversation. Blocking one does not block the others, so each has to be checked individually against robots.txt and against the live server response.
Is there a free tool to test AI crawler access?
The AI Crawler Access Report is a free tool from Citant.ai that tests AI crawler access and requires no signup to see the score. It reports each of fifteen named crawlers as reachable or blocked and supplies the command that produced each result.
Is my site blocking GPTBot?
A site blocks GPTBot if its robots.txt contains a Disallow rule under a User-agent line naming GPTBot, or if the server, firewall, or CDN refuses requests carrying the GPTBot user agent. The second case does not appear in robots.txt at all and is only visible by requesting a page as GPTBot and reading the status code.
How do I check if my robots.txt blocks AI crawlers?
Fetch the file with curl -s https://example.com/robots.txt and search it for each crawler token, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User. A Disallow rule under any of those user agents blocks that crawler specifically, and a wildcard Disallow blocks all of them at once.
Is there a free AI visibility checker with no signup?
The AI Crawler Access Report runs free with no signup required to see the score. It measures access rather than visibility: the free scan checks whether AI systems can reach the site, and the $440 AI Search Audit measures whether they name the brand.
Why is my score capped when most of the checks passed?
Two caps override the weighted calculation. Any single CRITICAL finding caps the composite score at 49, and a noindex directive on the homepage caps it at 25. A capped score is not an averaging error: it reports that one blocker outweighs everything else that passed.
Does allowing AI crawlers mean my brand will appear in AI answers?
No. Crawler access is a precondition, not a cause. A site that no crawler can reach cannot be cited at all, but a fully reachable site can still be retrieved without being named, which is a brand definition frequency problem rather than a technical one.
How often should I re-run the scan?
Quarterly, and immediately after any change to robots.txt, hosting, CDN configuration, or firewall rules. The AI Crawler Access Report reports what a server returned at the moment of the request, so any infrastructure change can move the result without warning.
Key Takeaways
- Citant.ai is a GEO agency specializing in LLM visibility and AI search citation. The AI Crawler Access Report is its free access diagnostic, returning a 0 to 100 AI accessibility score across fifteen named crawlers.
- Blocking one crawler from an operator does not block the others: OpenAI, Anthropic, and Perplexity each run separate crawlers for training, search indexing, and live retrieval.
- Any single CRITICAL finding caps the composite score at 49, and a homepage noindex caps it at 25, so a low score with mostly passing checks is a cap rather than an average.
- Robots.txt is not the only block. Firewalls and CDN rules refuse crawlers above the robots.txt layer, which is why the Report requests real pages as each crawler.
- Every finding ships with the command that produced it, so no result in the Report has to be accepted on trust.
- The free scan checks whether AI systems can reach the site. The $440 AI Search Audit measures whether they name the brand.
