Cloudflare says "more than 20%" of web domains sit behind its network, so its settings shape how AI crawlers experience a large slice of the web. Since 2025, Cloudflare has steadily added controls that let site owners block, allow or charge AI bots, and in 2026 it rebuilt those controls around what each bot actually does.
That is good news for publishers who want control, but it creates a new SEO risk. A single toggle can quietly stop the crawlers that feed ChatGPT search, Perplexity or even Google Search, and many site owners never look at the setting again after onboarding.
This guide explains the current Cloudflare options as of October 2026, from AI Crawl Control and the Search/Agent/Training categories to Disallow AI Training, Bot Preference Sync and pay per crawl, and shows how to configure them without losing AI search visibility.
Key Takeaways
- AI Crawl Control (formerly AI Audit) is available on all Cloudflare plans and shows which AI crawlers visit, what they request and whether they follow robots.txt.
- Since July 2026, all customers can manage AI traffic by purpose: Search, Agent and Training.
- From 15 September 2026, new domains that say they monetise with ads get Training and Agent bots blocked by default on pages with ads; Search stays allowed.
- For mixed-use crawlers such as Googlebot, Bingbot and Applebot, choose "Disallow AI Training" rather than "Block" if you want to stay in search.
- Pay per crawl is still a closed/private beta; most sites should focus on access settings first.
How Cloudflare’s AI controls evolved
| Date | Change | Source |
|---|---|---|
| 1 July 2025 | First "Content Independence Day": one-click Block AI Bots option and the pay per crawl private beta | Cloudflare blog |
| 28 August 2025 | AI Audit relaunched as AI Crawl Control | Cloudflare docs |
| 1 July 2026 | Second Content Independence Day: Search, Agent and Training categories for all customers, including Free | Cloudflare blog |
| 21 August 2026 | Bot Preference Sync announced to keep robots.txt aligned with dashboard settings, on every plan | Cloudflare blog |
| 15 September 2026 | Disallow AI Training, "Accountable" crawler designation, Block AI Bots and Managed robots.txt being replaced, new ad-page defaults for new domains | Cloudflare blog |
If you configured "Block AI Bots" in 2025 and never revisited it, your site may be running on settings Cloudflare is now phasing out. It is worth reviewing the Security settings for each zone.
What AI Crawl Control does
AI Crawl Control is Cloudflare’s dashboard for AI traffic. It requires no configuration and gives you:
- Visibility: requests over time by crawler, operator and category, data transfer per crawler, status codes and most-requested paths.
- Per-crawler actions: Allow, Block, or (for beta participants) Charge. Blocking creates or updates a WAF custom rule on your zone.
- robots.txt tracking: a "Robots.txt violations" view and an option to enforce robots.txt for crawlers you allow.
- Custom block responses (paid plans): return 403 Forbidden or 402 Payment Required with a plain-text message.
- Referral analytics (paid plans): which AI platforms, such as chatgpt.com, send visitors back.
One limitation to know: according to the Manage AI crawlers documentation, the Free plan identifies crawlers by user agent only, while paid plans add Bot Management detection. Combine it with your own log review (see our guide to AI bots in server logs) and verify crawlers before acting on surprising numbers.
The Search, Agent and Training model
In July 2026 Cloudflare moved from a simple block-or-allow switch to three purpose-based categories. Search bots collect or index content to answer questions later; Agent bots act in real time for a person (chat fetchers and browser agents); Training bots collect content to train or fine-tune models.
“Site owners should expect to get referral traffic or other equitable compensation as a result.”
— Cloudflare, Your site, your rules: new AI traffic options for all customers
That quote refers to Search bots, and it captures the trade-off: search-type access is what earns citations and referral clicks, while training access generally does not send traffic back directly.
Recommended settings for most businesses
| Category | Typical crawlers | Visibility impact if blocked | Suggested default |
|---|---|---|---|
| Search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | You can disappear from AI search answers and citations | Allow |
| Agent | ChatGPT-User, Claude-User, Perplexity-User, browser agents | Assistants cannot open your page when a user asks about you | Allow for most commercial sites; consider limits for ad-funded content |
| Training | GPTBot, ClaudeBot and similar | No direct citation impact documented by operators | Business decision: Allow or Disallow AI Training |
| Mixed-use (Search + Training) | Googlebot, Bingbot, Applebot | Blocking also blocks search crawling | Use Disallow AI Training, never Block, if you want search traffic |
Disallow AI Training vs Block: the setting that matters most
The September 2026 update made one change that SEOs must understand. Cloudflare’s "Block" and "Block on pages with ads" settings now apply to mixed-use crawlers too, which means they can block search crawling. The new Disallow AI Training option instead writes a no-training preference into robots.txt, so crawlers Cloudflare designates as "Accountable" can keep indexing for search. Cloudflare lists Apple, Google and Microsoft as Accountable, and also categorises Amazon, Anthropic, Meta and OpenAI that way.
There is a caveat for Bing: Cloudflare says Bing’s robots.txt support for no-training preferences is not live yet, with Microsoft targeting early 2027, so the setting will not convey that preference to Bing automatically until then. For Google, the established training control remains the Google-Extended robots.txt token, which Google says does not affect Search inclusion or ranking; our new explainer on Google-Extended covers it.
Bot Preference Sync and robots.txt
Bot Preference Sync replaces Managed robots.txt. It prepends a generated block to your existing robots.txt that reflects your dashboard choices, leaving your own rules in place. New customers get it on by default; publishers who select "I monetize from pages with ads" at onboarding get Training set to Disallow. After enabling it, open /robots.txt and read the generated section. Make sure it does not conflict with rules your SEO team maintains (our guide to AI crawlers and robots.txt explains the main user agents).
Pay per crawl: what it is and who it is for
Pay per crawl lets a site charge AI crawlers per request. Crawlers without payment intent receive an HTTP 402 Payment Required response with pricing; crawlers that agree to pay receive the content. Cloudflare acts as merchant of record. According to the launch post, publishers set one flat per-request price for the domain, can choose Allow, Charge or Block per crawler, and paying crawlers authenticate with signed requests (Web Bot Auth). It remains a closed beta, so most site owners cannot simply switch it on. Cloudflare also announced further monetisation products in late September 2026 (Pay Per Use and a Monetization Gateway beta), which are worth watching if you run a content business.
Step-by-step: configure Cloudflare without losing AI visibility
- Audit current settings. In each zone’s Security settings, note whether legacy Block AI Bots or Managed robots.txt is active.
- Review AI Crawl Control. Check which crawlers visit, which are blocked and whether any search crawlers are receiving 403s.
- Allow Search. Keep OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed if you want to be cited.
- Decide on Training. If you want to opt out, choose Disallow AI Training rather than Block, so mixed-use search crawlers keep working.
- Decide on Agents. For e-commerce, local services and B2B sites, allowing agents usually helps; ad-funded publishers may choose differently.
- Check robots.txt. Confirm the Bot Preference Sync block matches your intentions and coexists with your own rules.
- Test. Fetch key pages with crawler user agents and confirm 200 responses; Google’s AI features guidance reminds owners that crawling must be allowed by robots.txt and by any CDN or hosting infrastructure.
- Monitor monthly. Watch AI referrals, crawler status codes and citations after each change.
Common mistakes
- Using Block for mixed-use crawlers. It can remove you from search, not just from training.
- Assuming defaults are neutral. New ad-supported domains now start with Training and Agent blocked on ad pages.
- Blocking agents on product and booking pages. That stops assistants from checking your prices or availability when a customer asks.
- Forgetting subdomains and other zones. Settings are per zone; check blogs, shops and help centres separately.
- Overwriting robots.txt by hand. Coordinate with the generated Bot Preference Sync block instead.
- Not checking after Cloudflare updates. Features changed several times between 2025 and 2026; schedule a quarterly review.
Related Guides
- AI Crawler Load: Managing Server Costs From Bots
- Does Structured Data Help AI Search? What the Evidence Says
- ChatGPT Product Feeds: OpenAI’s Merchant Feed Specification Explained
Frequently Asked Questions
Does Cloudflare block AI crawlers by default?
Not for every site. Since 15 September 2026, new domains that select the ad-monetisation option get Training and Agent bots blocked by default on pages with ads, with Search allowed. Other new customers start without blocks or disallows. Check your own zone settings.
Will blocking AI bots in Cloudflare hurt my Google rankings?
It can if you use Block on mixed-use crawlers such as Googlebot. Use Disallow AI Training instead to express a no-training preference while keeping search crawling.
Is AI Crawl Control free?
Yes, it is available on all plans. Some features, such as custom block responses, referral analytics and stronger bot detection, need a paid plan.
Can I start charging AI crawlers today?
Only if you are accepted into the pay per crawl beta. Cloudflare describes it as closed/private beta and asks site owners to sign up.
Where do I change these settings?
In the Cloudflare dashboard, under each domain’s Security settings and the AI Crawl Control section.
Conclusion
Cloudflare now offers far more precise AI controls than a single on/off switch, but precision only helps if the settings match your goals. Allow search and agent access to the pages you want recommended, use Disallow AI Training rather than Block if you want to opt out of training, and review the configuration every quarter. Our technical SEO specialists and CDN SEO guidance can help you get this right, and you can request a custom quote for an AI crawler access review. Also see our new guide to managing AI crawler load.
References
- Cloudflare docs: AI Crawl Control
- Cloudflare docs: Manage AI crawlers
- Cloudflare docs: What is pay per crawl?
- Cloudflare blog: Introducing pay per crawl (Jul 2025)
- Cloudflare blog: Your site, your rules (Jul 2026)
- Cloudflare blog: Have it both ways (Sep 2026)
- Cloudflare blog: Say it once: Bot Preference Sync (Aug 2026)
- Google Search Central: AI features and your website
