There are two entirely different kinds of AI crawler and confusing them is the most consequential mistake in this whole area.
Training crawlers collect content to train future models. GPTBot. ClaudeBot. Google-Extended. Blocking them does not remove you from anything. It only means your content is not used in training.
Retrieval crawlers fetch pages in real time to answer a question a user has just asked, and they cite the source. OAI-SearchBot. PerplexityBot. Claude-SearchBot. Blocking these removes you from the answer entirely.
Which means: a site that blocked "AI bots" in 2023 to protect its content may have accidentally made itself invisible to ChatGPT search, Perplexity and Claude, and nobody noticed, because there is no error message and no report.
Check robots.txt. Today. It takes five minutes.
The two kinds, and why the distinction is everything
| Training crawler | Retrieval crawler | |
|---|---|---|
| What it does | Collects content to train a future model | Fetches your page to answer a question right now |
| Does it cite you? | No | Yes. That is the point. |
| If you block it | Your content is not used in training | You disappear from the answer |
| Examples | GPTBot, ClaudeBot, Google-Extended |
OAI-SearchBot, PerplexityBot, Claude-SearchBot |
| Should you block it? | A real business decision. | Almost never. |
Blocking a retrieval crawler is the modern equivalent of noindexing your homepage.
And people are doing it, at scale, by accident, because a security plugin, a CDN rule, or a well-meaning developer in 2023 added a blanket block on "AI bots" and nobody has looked since.
The crawlers, one by one
GPTBot — OpenAI. Training. Blocking it keeps your content out of future model training. It does not remove you from ChatGPT search.
OAI-SearchBot — OpenAI. Retrieval. This is the one that matters. It fetches pages to answer live queries in ChatGPT search, and it cites them. Block this and you are gone from ChatGPT.
ChatGPT-User — OpenAI. Fetches a page when a user explicitly asks ChatGPT to visit a URL. Blocking it means a user who pastes your link gets nothing.
ClaudeBot — Anthropic. Training.
Claude-SearchBot — Anthropic. Retrieval.
Claude-User — Anthropic. User-initiated fetch.
PerplexityBot — Perplexity. Retrieval, and Perplexity cites its sources prominently and sends real referral traffic. Blocking this is straightforwardly self-harm.
Google-Extended — Google. This one is genuinely confusing and it catches people out. It is not a crawler. It is a control that tells Google whether your content may be used to train Gemini. Blocking Google-Extended does NOT remove you from AI Overviews, because AI Overviews are built from Google's ordinary search index, crawled by Googlebot. You cannot opt out of AI Overviews while remaining in Google Search. People believe you can, and you cannot, and this is the single most misunderstood fact in the whole area.
Bingbot — Microsoft. The ordinary search crawler, and it is the one that gates ChatGPT search, because ChatGPT retrieves from Bing's index. A site not in Bing is invisible to ChatGPT regardless of every other setting on this page.
Applebot-Extended — Apple. Training control for Apple Intelligence.
Amazonbot, Bytespider, CCBot — Amazon, ByteDance, Common Crawl. CCBot is worth a thought: Common Crawl feeds many downstream datasets, so blocking it has broad and hard-to-predict effects.
The decision your client actually has to make
And it is a real one, so present it as one rather than assuming the answer.
The case for allowing everything: you want to be cited, recommended and named. For almost every business that sells something, visibility is worth more than the content itself. A plumber, a law firm, a SaaS company, an agency: all of them benefit from being in the answer.
The case for blocking the training crawlers: you are a publisher, and your content IS the product. A news organisation, a research house, a stock library, a paid-content business. You are being used to train a system that will eventually substitute for you, for free. That is a legitimate grievance and blocking is a legitimate response.
The case for blocking retrieval crawlers: there is almost none. These cite you and send traffic. Blocking them is choosing not to appear.
The recommendation for almost every commercial client:
# Allow retrieval. This is how you get cited.
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
# Training: a business decision. Default for most is allow.
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
And if the client is a publisher who wants out of training, block the training crawlers and keep the retrieval crawlers open. That is the nuanced position and almost nobody realises it is available.
How to actually check, in five minutes
Read yourdomain.com/robots.txt with your own eyes. Not through a tool. Read it.
Look for a blanket User-agent: * with a Disallow: /, which blocks everything, and which has ended more launches than every other technical error combined.
Look for AI crawler names, and check whether anybody blocked them in 2023 and forgot.
Check your CDN and your security plugin. Cloudflare has a one-click "block AI bots" toggle. Somebody may have clicked it. It does not appear in robots.txt and it does not tell you it is on. This is the trap that catches the most sites and it is completely invisible from the outside.
Check your server logs for the user agents above. Are they actually visiting? If not, something is blocking them.
And check whether the site is in Bing at all, because if it is not, none of the rest matters for ChatGPT.
→ Technical SEO · The best SEO tools
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content to train future models. OAI-SearchBot fetches your page in real time to answer a live query in ChatGPT search, and it cites you. Blocking GPTBot keeps you out of training. Blocking OAI-SearchBot removes you from ChatGPT entirely.
Should I block AI crawlers?
Almost never the retrieval crawlers, because they cite you and send traffic, and blocking them is choosing not to appear. Blocking the training crawlers is a legitimate decision if your content is your product, such as a publisher or research business.
Does blocking Google-Extended remove me from AI Overviews?
No, and this is the most misunderstood fact in the area. `Google-Extended` is a training control, not a crawler. AI Overviews are built from Google's ordinary search index, crawled by Googlebot. You cannot opt out of AI Overviews while remaining in Google Search.
How do I know if I am blocking AI crawlers?
Read `robots.txt` with your own eyes, then check your CDN and security plugin. Cloudflare has a one-click "block AI bots" toggle that does not appear in `robots.txt` and does not tell you it is on. That is the trap that catches the most sites.
Which crawler matters most?
Bingbot, indirectly. ChatGPT search retrieves from Bing's index, so a site absent from Bing is invisible to ChatGPT regardless of every other setting.
Can I allow retrieval but block training?
Yes, and almost nobody realises it is available. Block `GPTBot` and `ClaudeBot`, and allow `OAI-SearchBot`, `PerplexityBot` and `Claude-SearchBot`. That is the correct position for a publisher who wants citations without feeding the training set.
Does PerplexityBot send real traffic?
Yes. Perplexity cites its sources prominently and produces genuine referral clicks, which makes blocking it straightforwardly self-defeating.
Still Have Questions on Your Mind?
Contact our support team and we’ll guide you every step of the way.