What it checks
The Cyberbard AI Crawler Checker is a free tool that tests whether 17 AI and search crawlers, including GPTBot, ClaudeBot and PerplexityBot, are allowed by a site's robots.txt and can actually load the page when they visit.
robots.txt rules, per bot
Which group applies to each crawler (its own or the * group), the Allow and Disallow rules involved, and whether the exact path you entered is allowed. Follows RFC 9309, including * and $ wildcards.
Live test with real user agents
We request your page as GPTBot, ClaudeBot, PerplexityBot and the others. A 403, 429 or challenge page means your firewall or CDN refuses them, even if robots.txt says yes.
Training vs search vs live fetchers
Bots are grouped by what they do: collecting training data, building an AI search index, or fetching a page because a user just asked about it.
noai, noindex and X-Robots-Tag
Meta robots tags and X-Robots-Tag headers that tell crawlers not to index, follow, or use your content for AI.
robots.txt generator
Pick a policy (allow everything, allow AI search but not training, block AI) or toggle bots one by one, then copy a ready-made robots.txt block.
Sitemap and crawl-delay
Sitemap lines declared in robots.txt and any Crawl-delay that slows crawlers down.
How to use the AI Crawler Checker
Enter a domain or page
Type example.com to test the homepage, or paste a full URL to test a specific page against your rules.Read the verdict per bot
Allowed, partial, blocked by robots.txt, or "hidden block": allowed by robots.txt but refused by your server. Open a row to see the rule that decided it.Choose a policy and copy the rules
Use the generator to build the robots.txt block that matches what you want, and paste it into the robots.txt at your site's root.Fix firewall blocks separately
A hidden block isn't fixed in robots.txt: allow the bot in your CDN or WAF settings (for example Cloudflare's bot settings), then run the check again.What this tool does, and what it doesn't (yet)
So you know exactly what the result covers, and where to look for the rest.
What it does
- Parse robots.txt for 17 crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Googlebot, Bingbot, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and DuckAssistBot
- Apply RFC 9309 group rules: a group that names a bot replaces the * group for that bot
- Check the exact path you entered against Allow and Disallow rules, including * and $ wildcards
- Request your page with each bot's published user agent and detect 401, 403, 429 and bot-challenge pages
- Compare those answers with a normal browser-like request, so a site that blocks everyone isn't reported as blocking AI only
- Read meta robots, per-bot meta tags and X-Robots-Tag headers (noindex, nofollow, noai, noimageai, nosnippet)
- Generate a robots.txt block for a policy you choose, bot by bot
Not covered (yet), and what to do instead
Requests come from our servers, not from OpenAI's, Anthropic's or Google's IP ranges. Firewalls that allow or block by IP can treat the real bots differently than our test.
Instead: Check your CDN or WAF logs for the real bots' requests, or the verified-bot settings in your CDN.
We test one page per run, not every path your Disallow rules cover.
Instead: Run the check on your most important pages one by one.
We can't see whether a bot has actually visited you or how often.
Instead: Your server or CDN access logs show real crawler visits.
Pages behind a login or paywall can't be tested.
Instead: Test the public pages you want AI to read.
We don't change anything on your site.
Instead: Paste the generated rules into your robots.txt yourself, or hand them to your developer.
Google-Extended and Applebot-Extended aren't crawlers, only robots.txt tokens, so they can't be tested live.
Instead: Their robots.txt status is still checked.
Why AI crawler access matters
An AI crawler is the program an AI company uses to download web pages. Some collect training data, some build the search index that AI assistants quote from, and some fetch a single page the moment a user asks about it. If none of them can read your page, AI tools can only describe you from what other sites say.
Three kinds of AI crawler
- Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider) gather content that may be used to train models. Blocking them is a legitimate choice if you don't want your content used that way.
- AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, and classic search crawlers like Googlebot and Bingbot) build the indexes that answers with citations are drawn from. Blocking these is what usually removes you from AI answers.
- User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) load a page because a person asked the assistant to look at it. Vendors document these separately, and some say that robots.txt rules may apply to them differently than to automated crawling, so check each vendor's documentation.
A common policy is therefore "allow AI search, block training": you stay citable without volunteering your content for model training. The generator above writes exactly that block for you.
robots.txt isn't the whole story
robots.txt is a request, not a lock: it tells well-behaved crawlers what you'd like. The real lock is your server. Many sites run a CDN or firewall (Cloudflare, Akamai, Sucuri, a hosting provider's bot filter) that blocks "unknown bots" by default or through a one-click "block AI bots" setting. The result is a site whose robots.txt welcomes GPTBot while the firewall answers it with a 403 or a challenge page. Nothing in robots.txt reveals that, which is why this checker also makes a live request with each bot's user agent.
Common mistakes
- A blanket
User-agent: */Disallow: /left over from a staging site, which blocks every crawler that has no group of its own. - Copying a block list from a blog post that includes search crawlers (OAI-SearchBot, PerplexityBot) when the intent was only to stop training.
- Assuming the
*group still applies to a bot that has its own group. Under RFC 9309 it doesn't: a group that names GPTBot completely replaces*for GPTBot. - Firewall rules nobody remembers enabling, the source of most "hidden blocks".
noindexornoaimeta tags inherited from a template.
How to fix what this check finds
- Blocked by robots.txt: edit the group that applies (the checker shows which one) or paste the generated block. robots.txt must sit at the root of the host:
https://example.com/robots.txt. - Hidden block: allow the bot in your CDN or firewall, then re-run the check. In Cloudflare, look at the bot and AI crawler settings for your zone.
- Partial: decide whether the disallowed paths are ones AI should read. Blocking
/adminis fine; blocking/blogprobably isn't.
How this checker differs from a robots.txt tester
An honest comparison, including what we don't do.
Parses robots.txt rules
Covers AI training, AI search and user-triggered bots
Live request with each bot's real user agent
Detects firewall / CDN blocks robots.txt can't show
Checks meta robots and X-Robots-Tag (incl. noai)
Generates robots.txt for a policy you choose
Tests from the bots' real IP ranges
Frequently asked questions
Should I block GPTBot?
It depends on what you want. GPTBot collects content that OpenAI may use for training. Blocking it does not stop ChatGPT search from citing you, because that uses OAI-SearchBot. If you want to be cited but not used for training, block GPTBot and allow OAI-SearchBot; the generator's "Allow AI search, block training" policy does that.
What is the difference between training bots and AI search bots?
Training bots gather pages that may go into future models. AI search bots build the index an assistant searches when it answers with links. User-triggered fetchers load one page because a user asked about it. Blocking search bots and user fetchers is what usually removes you from AI answers.
robots.txt allows the bot, but the result says "hidden block". Why?
Your server refused our request made with that bot's user agent (HTTP 401, 403, 429 or a bot-challenge page) while it answered a normal browser-like request. That is almost always a CDN or firewall rule. Allow the bot in your CDN/WAF settings; changing robots.txt won't help.
Does blocking Google-Extended remove me from Google Search or AI Overviews?
Google documents that Google-Extended does not affect inclusion or ranking in Google Search; it controls whether content crawled by Googlebot may be used for Gemini models. AI Overviews are part of Search and follow Googlebot's rules, so to stay out of them you would have to restrict Googlebot itself.
Why might my results differ from what the real crawlers experience?
We send each bot's user agent from our own servers. Firewalls that verify bots by IP address (or allow "verified bots" only) may treat our request and the real crawler differently. If you see a block here but your CDN logs show the real bot getting through, trust the logs.
Can this tool see my server logs or tell me how often bots visit?
No. It only sees what any visitor sees: your robots.txt, your page and your server's responses. Your hosting or CDN access logs are the source for real visit counts.
Do AI crawlers respect robots.txt?
The major vendors (OpenAI, Anthropic, Google, Perplexity, Apple, Microsoft) publish their crawler names and state that their automated crawlers follow robots.txt. robots.txt is voluntary, though, and some crawlers ignore it, which is one reason sites also use firewall rules.
Where does robots.txt have to be?
At the root of each host, e.g. https://example.com/robots.txt. A subdomain (blog.example.com) needs its own file. The checker always reads the robots.txt of the host you enter.
Is it really free? Are there limits?
Yes, it's free and needs no account. To keep it fast for everyone, each visitor can run 20 checks per hour. If you hit the limit, the page tells you when you can try again.
Do you store the URL I check or the results?
No. The address and the results are only shown in your browser and are gone when you leave the page. To enforce the hourly limit we store a one-way hash of your IP address with a counter, nothing else.
What does the full Cyberbard report add?
This tool checks one thing. The full report asks AI the questions your customers ask, shows which competitors it recommends instead of you, scores 50+ factors across your site and the wider web, and writes a Fix plan for your developer plus a Growth plan for the marketing side. You get 3 free reports when you sign up, with no card and no subscription.
More free GEO tools
llms.txt Generator
Validate an existing llms.txt or generate a draft from your sitemap and navigation, with every link checked and an editor to copy or download the file.
Open the toolSchema Checker for AI
See whether your structured data tells AI who you are: Organization and sameAs completeness, key fields per type, and ready-to-paste JSON-LD for what is missing.
Open the toolAI Citability Checker
Score any page paragraph by paragraph for how easily AI can quote it, with the reasons behind each score and what to change.
Open the toolLast updated