A request that says it is GPTBot is not necessarily GPTBot. The user-agent header is just text, and any scraper can copy it. Some scrapers pretend to be well-known AI crawlers precisely because many sites allow them through. Others do the opposite and hide behind an ordinary browser user agent after being blocked.
If you make decisions about AI crawlers (allowing OAI-SearchBot so you appear in ChatGPT search, blocking training crawlers, or reporting AI crawl volume to your boss), you need to know which requests are genuine. Get it wrong and you either let impostors drain your server or block the real crawler that would have cited your content.
This guide explains how spoofing works, how to verify GPTBot, ClaudeBot, PerplexityBot and Google's crawlers using published IP ranges and DNS, and how to block impostors safely.
Key Takeaways
- User agents can be faked in seconds; IP verification is what proves a request’s origin.
- OpenAI, Anthropic and Perplexity publish JSON lists of their crawler IP addresses; Google supports both reverse DNS checks and published IP lists.
- Block on "claimed AI user agent + IP not in the official list", never on the user agent alone.
- Anthropic recommends using its IP list to verify, not as an opt-out method; use robots.txt for opting out.
- Cryptographic request signing (Web Bot Auth) is emerging as a stronger way to identify agents and crawlers.
Why AI user agents get spoofed
There are two kinds of deception to watch for. The first is impersonation: a scraper sends GPTBot or ClaudeBot in its user agent hoping to slip past rules that allow those crawlers. The second is evasion: a crawler that has been blocked under its declared name returns with a generic browser user agent.
The second pattern is documented. In August 2025, Cloudflare published an analysis titled Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives. Cloudflare reported that when its test sites blocked Perplexity’s declared crawlers, it observed requests using a generic Chrome-on-macOS user agent from IP addresses outside Perplexity’s published ranges. Cloudflare estimated the declared Perplexity-User traffic at about 20–25 million daily requests and the undeclared traffic at about 3–6 million. These are Cloudflare’s findings from its own network and tests, so treat them as one company’s analysis rather than an industry-wide measurement. The same post described OpenAI’s crawlers as stopping when disallowed in Cloudflare’s tests.
The lesson for site owners is not about any single company. It is that you need two layers: verification of declared bots, and behavioural bot detection for undeclared ones.
How each operator lets you verify its crawlers
| Operator | Crawlers | Verification method | Published list |
|---|---|---|---|
| OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot | Match source IP against the published JSON for each bot | openai.com/gptbot.json, /searchbot.json, /chatgpt-user.json, /adsbot.json |
| Anthropic | ClaudeBot, Claude-SearchBot, Claude-User | Match source IP against Anthropic’s list | claude.com/crawling/bots.json |
| Perplexity | PerplexityBot, Perplexity-User | Match user agent and IP in your WAF | perplexity.com/perplexitybot.json, /perplexity-user.json |
| Googlebot and other Google crawlers and fetchers | Reverse DNS plus forward DNS, or match against published IP lists | common-crawlers.json, special-crawlers.json, user-triggered-fetchers.json and others |
Sources: OpenAI crawler overview, Anthropic’s crawler help article, Perplexity’s bot guide and Google’s verification guide. Lists change, so automate downloads rather than copying ranges into a config once.
“You can verify if a request to your server really is from Google.”
— Google Search Central, Verifying Googlebot and other Google crawlers
Step-by-step: verifying a suspicious request
- Pull the request from your logs. Note the IP address, user agent, timestamp and URL.
- Identify the claimed operator. Match the user-agent token (GPTBot, ClaudeBot, PerplexityBot, Googlebot and so on).
- Check the official IP list. Download the operator’s current JSON file and test whether the IP falls inside any listed CIDR range. A small script can do this for thousands of IPs at once.
- For Google, you can also use DNS. Run a reverse lookup (
host 66.249.66.1), confirm the hostname ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that hostname and confirm it returns the original IP. - Classify the result. Verified, impostor (claimed AI user agent but IP not listed), or unknown (generic user agent with bot-like behaviour).
- Act on impostors only. Block or challenge them at the firewall or CDN and leave verified crawlers alone.
- Refresh lists automatically. Schedule a daily or weekly job to re-download the JSON files so new ranges are not blocked by mistake.
If you want to see the whole picture first, start with our new guide to finding AI bots in your server logs, then run verification on the subset that claims to be an AI crawler.
Blocking impostors without blocking real crawlers
The safest rule combines both signals. In plain language: "if the user agent claims to be an AI crawler and the IP is not in that operator’s published list, block or challenge it." Perplexity’s documentation explicitly advises using both user-agent matching and IP verification in WAF rules.
| Rule type | Example | Risk |
|---|---|---|
| User agent only | Block anything containing "GPTBot" | Blocks the real crawler; impostors simply change their user agent |
| IP only | Block a list of IP ranges | Ranges change; Anthropic notes IP blocking prevents bots from reading robots.txt |
| User agent + IP mismatch | Block "GPTBot" from IPs outside gptbot.json | Low risk, if lists are refreshed |
| Behavioural detection | CDN bot scores, rate anomalies, challenge pages | Catches undeclared crawlers; needs tuning to avoid blocking real users |
| Signed requests | Accept agents that sign with Web Bot Auth | Strong identity, but not yet universal |
Remember that verification and opting out are different jobs. Anthropic says its IP list should be used to verify that a crawler comes from Anthropic, and that blocking IPs is not a reliable opt-out because it stops the bots reading your robots.txt. To opt out of training or search crawling, use robots.txt rules for the specific user agent; our guide to AI crawlers and robots.txt explains the options, and the Google-Extended explainer covers Google’s training control.
Cryptographic verification: Web Bot Auth
IP lists are a stop-gap. A stronger approach is for bots to sign their requests. Cloudflare’s pay per crawl announcement (July 2025) describes Web Bot Auth: crawlers create an Ed25519 key pair, publish the public key, and sign each request using HTTP Message Signatures, which makes impersonation much harder. Cloudflare’s August 2025 analysis also noted that ChatGPT Agent signs its requests this way. If your CDN supports verifying signed agents, enable it (Cloudflare users can start with our explainer on Cloudflare AI Crawl Control); it complements, rather than replaces, IP checks. For agent traffic in particular, see our new guide on detecting AI agent visits.
Common mistakes
- Blocking by user agent alone. It punishes the honest crawler and barely slows down impostors.
- Hard-coding IP ranges. Operators add ranges; stale lists cause false blocks.
- Treating user-triggered fetchers like crawlers. ChatGPT-User and Perplexity-User act for a person, and their operators say robots.txt may not apply in the same way.
- Using IP blocks as your opt-out. Use robots.txt for opting out; use IP verification to enforce rules against impostors.
- Forgetting CDN and host settings. A firewall rule at the host can override what you set in robots.txt. Google’s AI features guidance says crawling must be allowed by robots.txt "and by any CDN or hosting infrastructure". If impostors are driving up costs, our AI crawler bandwidth guide covers rate limiting and caching.
- Reporting raw user-agent counts. Unverified numbers can overstate AI crawling significantly.
Related Guides
- Cloudflare’s AI Crawler Controls and Pay Per Crawl Explained
- AI Crawler Load: Managing Server Costs From Bots
- Does Structured Data Help AI Search? What the Evidence Says
Frequently Asked Questions
How do I know if GPTBot is real?
Check whether the request’s IP address falls within the ranges OpenAI publishes for GPTBot (openai.com/gptbot.json). A matching user agent from an unlisted IP is not genuine.
Does reverse DNS work for OpenAI or Anthropic crawlers?
Their documentation points to published IP lists for verification. Reverse and forward DNS verification is the method Google documents for its own crawlers.
Should I block AI crawlers by IP address?
Use IP lists to verify and to block impostors. To opt out of a genuine crawler, Anthropic and others recommend robots.txt, because IP blocks stop crawlers from reading your robots.txt rules.
What is Web Bot Auth?
A way for bots to cryptographically sign their HTTP requests so sites can confirm who sent them. Cloudflare supports it and reports that ChatGPT Agent signs its requests.
How often do crawler IP lists change?
Operators do not commit to a schedule, so automate a regular download and re-check before blocking ranges.
Conclusion
Verification is what turns AI crawler management from guesswork into policy. Use official IP lists and DNS checks, block only proven impostors, and keep real search crawlers able to reach the pages you want cited. If you would like help configuring rules in your CDN or firewall, our technical SEO team can review your set-up, and you can request a custom quote for an AI crawler audit. For background on bots in general, Semrush’s guide to bot traffic is a useful primer.
References
- Google Search Central: Verifying Googlebot and other Google crawlers
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web?
- Perplexity: Perplexity crawlers
- Cloudflare: Perplexity is using stealth, undeclared crawlers (Aug 2025)
- Cloudflare: Introducing pay per crawl (Jul 2025)
- Semrush: Bot traffic definition and best practices
