Analytics tools such as GA4 are built to measure people, not crawlers. When GPTBot, ClaudeBot or PerplexityBot visit your site, they normally do not run your tracking script, so they never appear in your marketing dashboards. The only complete record of what AI systems request from your site is your server (or CDN) log.
That record is surprisingly valuable. It tells you which AI companies are collecting your content, which pages they care about, whether they hit errors, and how often a real user in ChatGPT, Claude or Perplexity caused a live fetch of one of your pages. That last signal is about as close as you can get to seeing AI "demand" for your content before any click happens.
This practical guide shows how to find AI bots in your logs, how to separate training, search and user-triggered traffic, and how to turn raw lines into decisions about crawl access and content.
Key Takeaways
- Server and CDN logs are the only place you see AI crawler requests in full; GA4 and similar tools mostly miss them.
- AI user agents fall into three groups: training crawlers, search/index crawlers and user-triggered fetchers. Each answers a different business question.
- User-triggered fetches (ChatGPT-User, Claude-User, Perplexity-User) are a useful proxy for pages people are asking assistants about.
- Always verify that a request really comes from the claimed operator before drawing conclusions, because user agents are easy to fake.
- Review AI bot logs monthly alongside crawl errors, robots.txt changes and AI referral traffic.
Where to find your logs
Where your logs live depends on your hosting set-up. Shared hosts usually offer raw access logs in the control panel. VPS and dedicated servers write them to files such as Apache's access.log or Nginx's access.log. If you sit behind a CDN, the CDN log is often more complete because some requests never reach your origin. Our general guide to log file analysis covers formats and retention; here we focus on AI traffic specifically.
If you use Cloudflare, its AI Crawl Control analytics already group AI requests by crawler, operator and category, and show requests over time, data transfer, status codes and popular paths (referral analytics require a paid plan). That is a fast way to start, but raw logs still give you more flexibility.
The AI user agents to look for
The most important step is classifying bots by purpose. The table below is built from each operator's own documentation as of October 2026. Versions in user-agent strings change, so match on the token (for example GPTBot) rather than the full string.
| Token in user agent | Operator | Purpose (per operator docs) | Group |
|---|---|---|---|
| GPTBot | OpenAI | Collects content that may be used to train foundation models | Training |
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT search results | Search |
| ChatGPT-User | OpenAI | Visits pages when a user asks ChatGPT or a Custom GPT to; not used for automatic crawling | User-triggered |
| OAI-AdsBot | OpenAI | Checks landing pages submitted as ads | Ads |
| ClaudeBot | Anthropic | Collects web content for model training | Training |
| Claude-SearchBot | Anthropic | Indexes content to improve search results | Search |
| Claude-User | Anthropic | Fetches pages when a Claude user asks a question | User-triggered |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity results; not used for training | Search |
| Perplexity-User | Perplexity | Visits pages in response to user questions | User-triggered |
| Googlebot (and other Google crawlers) | Search crawling; Google-Extended is a robots.txt token only, with no separate user agent | Search (mixed use) |
Note the Google row. Google states that Google-Extended has no separate HTTP user agent; it is a control token in robots.txt. You will never see "Google-Extended" in your logs, so you cannot measure Gemini training crawls separately. Our new explainer on Google-Extended goes deeper.
“Because these actions are initiated by a user, robots.txt rules may not apply.”
— OpenAI, Overview of OpenAI crawlers
That OpenAI note about ChatGPT-User matters for analysis. User-triggered fetchers behave differently from crawlers: Perplexity also says its user-triggered fetcher generally ignores robots.txt because a person requested the page. So if you see ChatGPT-User hits on a page you disallowed for GPTBot, that is expected behaviour, not necessarily a violation.
Step-by-step: filtering your logs for AI bots
- Export at least 30 days of logs. AI crawl patterns are bursty; a single day can mislead you.
- Filter by user-agent token. For example:
grep -Ei "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" access.log > ai.log. - Verify the IP addresses. Compare source IPs against each operator’s published ranges (OpenAI, Anthropic and Perplexity all publish JSON lists). Discard or flag mismatches. Our guide to verifying AI crawlers shows the method.
- Tag each line with a group. Add a column for Training, Search, User-triggered or Ads using the table above.
- Normalise URLs. Strip tracking parameters and group by template (product, category, blog, docs) so patterns are visible.
- Count requests, unique URLs and status codes per group and per template.
- Compare with your sitemap and analytics. Which important URLs are never fetched? Which pages get user-triggered fetches but little organic traffic?
- Record decisions and repeat monthly. Note robots.txt or firewall changes so you can see their effect in the next export.
Spreadsheet users can do all of this with a pivot table. Larger sites usually load logs into BigQuery, a log analyser or their observability platform. Semrush's log file analysis guide (September 2025) also notes that logs can reveal how AI bots such as ChatGPT-User interact with a site.
Turning the data into insights
Raw counts are not the goal. Use these questions to turn AI bot data into actions.
| Question | What to look at | Possible action |
|---|---|---|
| Can AI search crawlers reach my key pages? | OAI-SearchBot, Claude-SearchBot, PerplexityBot hits on money pages | Fix robots.txt, firewall or CDN blocks; improve internal links |
| What are people asking assistants about? | User-triggered fetches by URL | Strengthen and update those pages; add clear answers and data |
| Are bots wasting effort? | 404s, redirect chains, parameter URLs | Fix broken links, consolidate parameters, update sitemaps |
| Is anyone spoofing AI bots? | Claimed AI user agents from unlisted IPs | Block or challenge impostors at the firewall |
| Is training crawling worth the cost? | GPTBot and ClaudeBot volume and bytes | Decide on allow, limit or disallow (see our AI crawler bandwidth guide) |
| Did my content get picked up after an update? | Fetch dates after publishing or updating | Compare with AI citations and referral traffic |
Error rates deserve special attention. Vercel’s December 2024 analysis reported that around 34% of ChatGPT and Claude crawler fetches on its network hit 404 pages, often for outdated assets, compared with about 8% for Googlebot. If your logs show something similar, a clean-up of redirects and sitemaps is cheap and worthwhile.
Connecting logs to AI traffic and visibility
Logs show demand from bots; they do not show clicks. Combine them with referral data in GA4 (our guide on tracking AI traffic in GA4 explains the set-up) and with an AI visibility tracker. Pages that receive frequent user-triggered fetches but few referrals may be answering questions without earning the click, a pattern covered in our new piece on AI traffic that shows up as direct. A dashboard that combines all three sources is described in building an AI search dashboard in Looker Studio.
Common mistakes
- Trusting the user agent alone. Anyone can send "GPTBot" in a header. Verify against published IP ranges before reporting numbers.
- Treating all AI bots as one group. Blocking a training crawler and blocking a search crawler have very different consequences.
- Reading user-triggered fetches as crawling. ChatGPT-User or Perplexity-User hits reflect user questions, so they will not follow robots.txt the same way.
- Looking only at origin logs behind a CDN. Cached responses may never reach your server, so you undercount.
- Keeping logs for too short a time. Many hosts rotate logs after a few days; set retention to at least 30–90 days.
- Forgetting privacy. Logs contain user IP addresses. Limit access and follow your data protection obligations when storing them.
Related Guides
- Fake AI Bots: How to Verify GPTBot, ClaudeBot and PerplexityBot Requests
- Cloudflare’s AI Crawler Controls and Pay Per Crawl Explained
- AI Crawler Load: Managing Server Costs From Bots
Frequently Asked Questions
Why don’t AI bots show up in GA4?
GA4 relies on a JavaScript tag running in a browser. Most crawlers do not execute that tag, so their requests appear only in server or CDN logs.
What is the difference between GPTBot and ChatGPT-User?
According to OpenAI, GPTBot crawls content that may be used to train models, while ChatGPT-User visits pages when a user asks ChatGPT or a Custom GPT to, and is not used for automatic crawling.
Can I see Google-Extended in my logs?
No. Google says Google-Extended has no separate user agent; it is only a robots.txt token. Gemini-related crawling arrives under existing Google user agents.
How often should I analyse AI bot logs?
Monthly is a sensible rhythm for most sites, plus a check after major robots.txt, firewall or site changes.
Which AI bot metric matters most?
For most businesses, user-triggered fetches by URL are the most actionable, because they point to pages people are asking AI assistants about.
Conclusion
Your server logs are the closest thing to a ground-truth view of how AI systems use your content. Classify bots by purpose, verify them, and review the data every month next to your AI referrals. If you would like help setting this up, our technical SEO team can build an AI bot log review into your reporting, or you can request a custom quote.
References
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web?
- Perplexity: Perplexity crawlers
- Google Search Central: Google’s common crawlers
- Cloudflare: Analyze AI traffic (AI Crawl Control)
- Vercel: The rise of the AI crawler (Dec 2024)
- Semrush: What is a log file analysis? (Sep 2025)
