For most of the web’s history, the bots that mattered to a site owner were search engine crawlers, and their load was predictable. AI changed that. Training crawlers, AI search indexers and assistants fetching pages on demand now add a new layer of automated traffic, and on some sites it shows up as slower pages, higher hosting bills and surprise overage charges.
The answer is rarely "block everything". Many of these bots are the route by which your brand appears in ChatGPT, Claude or Perplexity answers. The goal is to measure the load, understand which bots cause it, and apply the lightest control that solves the problem.
This guide explains why AI crawlers can spike bandwidth, how to measure their real cost, and the options available in 2026: caching, rate limiting, crawl-delay, selective blocking and fixing the waste on your own site.
Key Takeaways
- AI crawlers often request long-tail and uncached pages, so they can cost more per request than human visitors.
- Measure first: requests, bytes transferred and origin hits per bot, verified against official IP lists.
- Caching and fixing crawl waste (404s, redirects, parameter URLs) usually cut load without reducing visibility.
- Rate limiting and Anthropic’s supported Crawl-delay are gentler than blocking; Google documents short-term 429/503 responses for Googlebot only.
- Block selectively: training crawlers and impostors first, search crawlers last.
Why AI bots can spike bandwidth
The clearest public example comes from the Wikimedia Foundation. As reported by TechCrunch in April 2025, Wikimedia said bandwidth for multimedia downloads from Wikimedia Commons had grown 50% since January 2024, driven mainly by automated scrapers rather than human readers. Bots accounted for 35% of pageviews but 65% of the most resource-intensive traffic. The reason is caching: popular pages are served from caches close to readers, while bots crawl less popular pages that must be fetched from the more expensive core data centre.
The same dynamic applies to ordinary business sites. Humans mostly read your top pages; crawlers walk your whole archive, including old tag pages, filtered category URLs and media files. Scale matters too. Vercel’s December 2024 analysis counted about 569 million GPTBot requests and 370 million Claude requests across its network in a single month, about 28% of Googlebot’s volume when combined with AppleBot and PerplexityBot. The same study found that roughly a third of ChatGPT and Claude fetches hit 404s, which is pure waste for both sides.
What drives AI crawler load
| Driver | Why it costs more | Typical fix |
|---|---|---|
| Long-tail and archive URLs | Low cache hit rate, so requests reach your origin | Longer cache TTLs, static caching of archives |
| Faceted and parameter URLs | Near-infinite URL combinations | Canonicals, robots.txt rules for parameters, fewer crawlable filters |
| Large media files | High bytes per request | Image optimisation, CDN delivery, hotlink rules |
| Broken and outdated URLs | Repeated 404s and redirect chains | Fix links, update sitemaps, clean redirects |
| Impostor bots | Unverified scrapers using AI user agents | IP verification and firewall rules |
| Uncached dynamic pages (SSR without caching) | Every hit triggers server rendering | Edge or full-page caching |
How to measure the real cost
- Pull 30 days of logs from your origin and your CDN.
- Identify AI bots by user agent (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User and others). Our guide to AI bots in server logs has the full list.
- Verify them against published IP lists so impostors are counted separately (see how to verify AI crawlers).
- Sum requests and bytes per bot, and split cache hits from origin hits.
- Check response times and 5xx errors during crawl bursts; correlate with CPU and database load.
- Estimate cost using your host’s bandwidth and compute pricing, and compare against AI referral traffic and citations.
If you are on Cloudflare, AI Crawl Control analytics show data transfer per crawler and over time, which makes the bytes question easy to answer. Other CDNs offer similar bot reports.
Your options, from lightest to heaviest
| Option | What it does | Effect on AI visibility | Notes |
|---|---|---|---|
| Fix crawl waste | Remove 404s, redirect chains, infinite parameters | Positive | Helps Googlebot too; see crawl budget guidance |
| Caching | Serve repeat bot requests from CDN or page cache | Neutral | Usually the biggest single saving |
| Crawl-delay | Asks a crawler to slow down | Neutral | Anthropic documents support; not universal |
| Rate limiting | Caps requests per IP or bot per period | Low risk if limits are generous | Return 429 rather than errors that look like outages |
| Disallow training crawlers | robots.txt rules for training user agents | No documented effect on AI search citations | Business and licensing decision |
| Block impostors | Firewall rules for unverified bots | Neutral | Do this first |
| Block search crawlers | Deny OAI-SearchBot, PerplexityBot and similar | Negative: you can lose AI citations | Last resort |
Crawl-delay and status codes
Anthropic’s crawler documentation says it supports the non-standard Crawl-delay directive, for example User-agent: ClaudeBot followed by Crawl-delay: 1. Other AI operators do not document Crawl-delay support on the pages we reviewed, so do not rely on it universally. For Google, the documented way to slow crawling in an emergency is to return 500, 503 or 429 responses briefly; Google then slows crawling across the hostname.
“We don’t recommend that you do this for a long period of time”
— Google Search Central, Reduce the Google crawl rate
Google warns that URLs returning these errors for several days may be dropped from the index, so treat it as a short-term emergency measure, not a policy. AI operators have not published equivalent guidance, so watch your logs to see how each crawler responds to 429s.
Selective blocking
If you decide to block, be precise. OpenAI’s crawler overview notes that OAI-SearchBot and GPTBot settings are independent, so you can allow search while disallowing training. Changes to search-related robots.txt rules can take about 24 hours to apply. Blocking at the firewall is also possible, but remember that undeclared crawlers exist: Cloudflare’s August 2025 analysis documented a crawler switching to a generic browser user agent after being blocked, which is why behavioural bot management matters on high-traffic sites. Our explainer on Cloudflare AI Crawl Control covers purpose-based controls, and the existing guide to AI crawlers covers robots.txt syntax.
Step-by-step plan for a site under load
- Confirm the problem with logs and hosting metrics, not assumptions.
- Block or challenge unverified bots claiming to be AI crawlers.
- Turn on or extend CDN caching for HTML and media, including archive pages.
- Fix the top 404s, redirect chains and parameter traps that bots are hitting.
- Add Crawl-delay for crawlers that support it, and rate limits for the rest.
- Decide on training crawlers as a business choice; keep search and user-triggered fetchers allowed for pages you want cited.
- Re-measure after two to four weeks and adjust.
Common mistakes
- Blocking every AI bot at once. It saves bandwidth but can remove you from AI answers.
- Using long-running 503s. Prolonged errors can harm Google indexing.
- Ignoring caching. Many "bot problems" are really cache configuration problems.
- Counting unverified traffic as AI crawlers. Impostors inflate the numbers and the costs.
- Leaving infinite URL spaces open. Faceted navigation and calendars can trap any crawler.
- Not linking cost to value. Compare crawl cost with AI referrals and citations before deciding.
Where this fits
Bandwidth management is part of a wider technical foundation for AI search. Make sure the pages bots do fetch are readable (see do AI crawlers render JavaScript?), hosting is fit for purpose (our guide to SEO hosting), and that your sitemap signals real updates (our new guide to freshness signals for AI search).
Related Guides
- Does Structured Data Help AI Search? What the Evidence Says
- ChatGPT Product Feeds: OpenAI’s Merchant Feed Specification Explained
- Should Your Business Publish an MCP Server?
Frequently Asked Questions
Are AI crawlers really increasing hosting costs?
For some sites, yes. Wikimedia reported a 50% increase in multimedia bandwidth since January 2024, driven mainly by scrapers. Impact varies widely, so measure your own logs.
Does Crawl-delay work for GPTBot?
OpenAI’s crawler documentation does not mention Crawl-delay. Anthropic explicitly supports it for its bots. Use rate limiting for crawlers that do not.
Will blocking GPTBot reduce my ChatGPT visibility?
OpenAI says GPTBot (training) and OAI-SearchBot (search) are controlled independently, so you can disallow GPTBot while allowing OAI-SearchBot for ChatGPT search.
Is rate limiting safe for SEO?
Generous limits that return 429 for short bursts are generally low risk. Avoid long periods of errors for Googlebot, which Google warns can lead to URLs dropping from the index.
What is the quickest win?
Caching. Serving repeat bot requests from a CDN or page cache often removes most of the origin load without changing what crawlers can access.
Conclusion
AI crawler load is real, but it is manageable. Measure it, cache aggressively, fix the waste and use precise controls rather than blanket blocks, so you protect both your server and your visibility in AI answers. Our page speed optimization team can tune caching and server performance, or you can request a custom quote for a crawler load review.
References
- TechCrunch: AI crawlers cause Wikimedia Commons bandwidth demands to surge 50% (Apr 2025)
- Vercel: The rise of the AI crawler (Dec 2024)
- Anthropic: Does Anthropic crawl data from the web?
- Google Search Central: Reduce the Google crawl rate
- OpenAI: Overview of OpenAI crawlers
- Cloudflare docs: Analyze AI traffic
- Cloudflare: Perplexity is using stealth, undeclared crawlers (Aug 2025)
