Open your server logs today and you will probably see a long list of AI bots: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Applebot and more. Some collect content to train AI models. Others index pages so AI assistants can cite and link to you. A few only visit when a real person asks an assistant a question. Treating them all the same is the most common, and most expensive, mistake site owners make.
The decision has become more urgent. Cloudflare, which sits in front of a large share of websites, announced in July 2025 that it was switching to blocking AI crawlers by default and, in September 2026, launched a setting that opts sites out of AI training without blocking search crawlers. Many site owners are now blocking AI bots without realising it, or blocking the wrong ones.
This guide explains the main AI crawlers, what each one does, the trade-offs of blocking or allowing them, and a simple decision process with ready-to-adapt robots.txt rules.
Key Takeaways
- Most major AI companies now run separate crawlers for model training and for search or user-requested fetching.
- Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) is a business decision that, according to their operators, does not remove you from those companies' search features.
- Blocking search and user crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User) can remove you from AI answers and referral traffic.
- Robots.txt is a request, not a lock. Check CDN and firewall settings too, because blocks there override what robots.txt says.
- Most businesses that want leads should allow AI search crawlers; publishers that rely on ad revenue have stronger reasons to restrict training.
The main AI crawlers in 2026
The table summarises the user agents documented by the major AI companies. Always check the vendor pages in the References section, because names and purposes change.
| Company | User agent | Purpose | Blocking it means |
|---|---|---|---|
| OpenAI | GPTBot | Collects content to make generative AI foundation models more useful and safe | Content should not be used for training |
| OpenAI | OAI-SearchBot | Surfaces websites in ChatGPT's search features | Your site may not appear in ChatGPT search answers |
| OpenAI | ChatGPT-User | Visits pages for certain user actions in ChatGPT and custom GPTs | ChatGPT cannot fetch your page when a user asks |
| Anthropic | ClaudeBot | Collects content that may improve Anthropic's models | Future pages excluded from training |
| Anthropic | Claude-SearchBot | Improves search result quality for Claude users | Content not indexed for Claude's search |
| Anthropic | Claude-User | Fetches pages when a Claude user asks a question | Claude cannot retrieve your pages for users |
| Perplexity | PerplexityBot | Surfaces and links websites in Perplexity search; not for foundation model training | Your site may not appear in Perplexity results |
| Perplexity | Perplexity-User | Visits pages to answer a user's question | Perplexity says this agent generally ignores robots.txt because a user requested it |
| Google-Extended (token, not a separate crawler) | Controls use of content for training Gemini models and grounding in some Google AI systems | No effect on Google Search inclusion or ranking | |
| Apple | Applebot-Extended (token) | Controls use of Applebot-crawled data for Apple's generative AI training | Apple says pages can still appear in its search results |
The Google-Extended distinction is important because many site owners fear that blocking it will hurt rankings. Google's documentation is unambiguous.
“Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.”
— Google Search Central, Google's common crawlers
Note what Google-Extended does not do: it does not stop your content appearing in AI Overviews or AI Mode, which are part of Google Search. Google's guidance for those features points to preview controls such as nosnippet, data-nosnippet and max-snippet instead. Our guides to the X-Robots-Tag and robots.txt vs meta robots explain how those controls work.
The case for blocking, and the case for allowing
Why some sites block
Publishers worry that AI systems use their work without sending traffic back. Cloudflare made this argument forcefully when it announced its default block on 1 July 2025, claiming that "with OpenAI, it's 750 times more difficult to get traffic than it was with the Google of old" and that the figure for Anthropic was 30,000 times. These are Cloudflare's own estimates, but they reflect a real concern for businesses that earn money from page views. Our analysis of AI traffic loss looks at this in more detail.
Server load is another reason. Aggressive bots can consume bandwidth and crawl budget, particularly on large sites.
Why most businesses should allow search crawlers
For a business that sells services or products, being cited in ChatGPT, Claude or Perplexity is free exposure at the moment someone is researching a purchase. Blocking search crawlers removes you from those answers and hands the recommendation to competitors. OpenAI also notes that changes to robots.txt can take around 24 hours to affect its search results, so mistakes are not always noticed quickly.
A decision framework
| Site type | Training crawlers | Search and user crawlers | Reasoning |
|---|---|---|---|
| Local service business | Allow (usually) | Allow | Wider awareness, little to lose from training use of marketing pages |
| B2B SaaS or professional services | Allow or selectively block | Allow | Buyers research with AI assistants; visibility drives pipeline |
| Ecommerce | Allow or selectively block | Allow | Product discovery and shopping features rely on search access |
| Ad-funded publisher | Often block | Decide case by case | Page views are revenue; consider licensing or pay-per-crawl options |
| Paywalled or premium content | Block | Allow only public teaser pages | Protect the product you sell |
Cloudflare's own September 2026 guidance, as reported by Semrush, follows similar logic: leave training allowed for sites that do not run ads, and disallow it for publishers that earn ad revenue from content.
Step-by-step: setting your AI crawler policy
- Find out who visits. Use log file analysis or your CDN's bot analytics to list AI user agents and how often they crawl. Google Search Console's crawl stats report covers Google's crawlers.
- Check existing blocks. Review robots.txt, CDN bot settings, firewall rules and any security plugins. A firewall block will not show up in robots.txt testing tools.
- Decide by purpose. Use the framework above to decide separately for training, search and user-request crawlers.
- Write explicit rules. Name each user agent rather than relying on wildcards.
- Protect private areas. Disallow account, cart, internal search and staging paths for all bots.
- Verify. Confirm bots are what they claim to be using the vendors' published IP ranges where offered, and watch logs after changes.
- Review quarterly. New agents appear regularly, and vendors change their documentation.
Here is an example robots.txt for a business that wants to appear in AI search but opt out of model training:
# Opt out of AI model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# Allow AI search and user-requested fetching
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
# Everyone: keep private areas out
User-agent: *
Disallow: /cart/
Disallow: /account/Remember that a crawler follows the most specific group that matches its name, so the * rules above do not apply to bots with their own group. Repeat private-path disallows inside each named group if needed.
Common mistakes
- Blocking every "AI" bot in one go. You lose AI search visibility along with training opt-out.
- Thinking Google-Extended controls AI Overviews. It does not; use snippet controls for that.
- Forgetting the CDN. Semrush notes that robots.txt-only audit tools will not detect blocks applied at Cloudflare's firewall.
- Assuming robots.txt is enforcement. It is a voluntary standard. Use firewall rules for bots that ignore it.
- Relying on llms.txt instead. An llms.txt file does not control crawling; robots.txt and server rules do.
- Setting and forgetting. New user agents appear and existing ones change purpose.
For a wider view of which assistants matter, see our guides to AI search engines and getting mentioned by Claude, and our checklist to optimise for AI agents.
Related Guides
- Wikidata and Wikipedia for Brands: Feeding the AI Knowledge Graph
- Content Chunking: Write Passages AI Can Quote
- How to Fix Wrong or Outdated Information About Your Brand in AI
Frequently Asked Questions
Will blocking GPTBot hurt my SEO?
No. GPTBot is OpenAI's training crawler and has nothing to do with Google rankings. Blocking it also does not remove you from ChatGPT search, which uses OAI-SearchBot.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode are part of Search; use nosnippet or max-snippet controls to limit how content is shown.
Do AI crawlers respect robots.txt?
The major vendors say their crawlers honour robots.txt. Perplexity states that its user-triggered agent generally does not, because a person requested the page. Use firewall rules where you need enforcement.
Is Cloudflare blocking AI bots on my site?
Possibly. Cloudflare moved to blocking AI crawlers by default in July 2025 and added training-only controls in 2026. Check the bot settings in your Cloudflare dashboard.
Should small businesses block AI crawlers?
Most should allow AI search crawlers so they can be recommended in AI answers. Whether to allow training crawlers is a judgement call; for typical marketing pages the risk is low.
Conclusion
AI crawler policy is no longer a yes-or-no question. Separate training from search, decide each on its business merits, write explicit rules and check that your CDN agrees with your robots.txt. For most businesses that means opting into AI search while making a deliberate choice about training.
Not sure what your site is currently blocking? Our technical SEO specialists can audit robots.txt, CDN and firewall settings and set a policy that protects your content without hiding you from AI search. Get a quote today.
References
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web?
- Perplexity: Perplexity crawlers
- Google Search Central: Google's common crawlers
- Apple: About Applebot
- Cloudflare: Content Independence Day, no AI crawl without compensation
- Semrush: Cloudflare's AI training block now spares Googlebot
