Most Search Console reports tell you what Google did with your pages: indexed them, ranked them, showed them for a query. The Crawl Stats report tells you something earlier in the chain: how Googlebot actually behaves on your server. How many requests it makes, how quickly your server answers, which files it fetches, and whether it is running into errors.
That makes it one of the best early-warning tools in technical SEO. Server slowdowns, robots.txt failures, DNS problems and sudden spikes in crawling of useless URLs often show up here before they hit rankings.
In this guide you will learn where to find the report, what every chart and table means, what "good" looks like, and a step-by-step routine for reviewing crawl stats each month.
Key Takeaways
- The Crawl Stats report shows Google's crawling history for your site: total requests, download size, average response time and host status.
- Google says the report is aimed at advanced users; sites under roughly 1,000 pages usually do not need to watch it closely.
- Host status covers robots.txt fetching, DNS resolution and server connectivity over the past 90 days.
- Breakdowns by response code, file type, purpose and Googlebot type show where crawling goes.
- Look for trends and step changes, not small day-to-day fluctuations.
Where to Find the Crawl Stats Report
In Google Search Console, open Settings in the left-hand menu, then click Open report next to Crawl stats. It is available for domain properties and for URL-prefix properties at the root level. Google launched the redesigned version of the report in November 2020.
“This report is aimed at advanced users.”
— Search Console Help, Crawl Stats report
That is a fair warning. If you run a small brochure site, you will get more value from the Page indexing report and the Performance report. For larger sites, ecommerce stores and publishers, Crawl Stats is essential.
The Main Charts Explained
| Metric | What it shows | What to watch for |
|---|---|---|
| Total crawl requests | All requests Googlebot made to your site, including resources and failed requests | Sudden drops (access problems) or spikes (new URL spaces) |
| Total download size | Bytes downloaded during crawling | Rising size without new content, such as bloated pages |
| Average response time | Average time to fetch a resource, excluding page rendering | Increases that coincide with fewer requests |
| Host status | Whether Google had availability problems in the last 90 days | Any red status, especially recent ones |
Response time deserves special attention. Google's crawl capacity depends partly on how your server copes; if responses slow down or errors rise, Google crawls less. Our guide to crawl budget explains this in detail.
Understanding Host Status
Host status is split into three categories:
- robots.txt fetching: whether Google could retrieve your robots.txt file. Google requests it frequently, and if it does not get a valid file or a 404, it can slow or stop crawling.
- DNS resolution: whether your DNS server recognised the hostname and responded.
- Server connectivity: whether your server responded fully, or timed out or refused connections.
Google's help page gives one concrete example of a threshold: a DNS failure rate above 5% on a given day is considered an issue for that category. Ideally, all three categories should be green. If you see recent problems, check with your hosting provider and read our guide to web hosting and SEO.
The Breakdown Tables
| Breakdown | Categories | How to use it |
|---|---|---|
| By response | 200, 301, 304, 404, 5xx, robots.txt not available and more | Most should be 200 or other "good" responses; investigate growing error groups |
| By file type | HTML, image, JavaScript, CSS, PDF, JSON, other | Spot crawling spent on unexpected files such as API endpoints |
| By purpose | Discovery (new URLs) or Refresh (known URLs) | Check that new content is being discovered after launches |
| By Googlebot type | Smartphone, Desktop, Image, Video, Page resource load, AdsBot, StoreBot and others | Confirm smartphone crawling dominates; investigate unusual AdsBot volume |
“Most responses should be 200 or other "Good"-type responses.”
— Search Console Help, Crawl Stats report
Click any row to see example URLs. These are samples, not complete lists, but they are often enough to identify a pattern such as a parameter or a broken template. If you need the complete picture, combine the report with server log file analysis.
A Monthly Crawl Stats Review (Step by Step)
- Check host status first. Any red or recent issue takes priority over everything else.
- Scan the three trend lines for step changes. Note the date of any big jump or drop and compare it with releases, migrations or hosting changes.
- Open the response breakdown. Look at the share of 404s, redirects and especially 5xx server errors. Click through to example URLs.
- Review file types. If JSON, JavaScript or "other" requests are growing fast, check whether bots are crawling APIs or infinite URL spaces.
- Compare discovery and refresh. After publishing a batch of content, discovery should rise. If it does not, check internal links and sitemaps, and see our guide to Discovered – currently not indexed.
- Check Googlebot types. Smartphone should usually lead because of mobile-first indexing.
- Record findings. Keep a short log with screenshots, because the report only covers a limited window.
Patterns and What They Usually Mean
- Requests drop while response time rises: server strain is causing Google to back off.
- Requests spike with no new content: a new parameter, faceted filter or calendar is creating crawlable URLs; this is often a sign of index bloat.
- Large share of 301s: internal links still point to redirected URLs.
- robots.txt not available: a firewall, CDN rule or misconfiguration is blocking or erroring on
/robots.txt.
Common Mistakes
- Chasing more crawl requests as a goal. More crawling is not better if it is spent on low-value URLs.
- Ignoring host status because rankings look fine today.
- Treating example URLs as complete lists.
- Blocking Googlebot by accident with bot-protection or rate-limiting rules at the CDN.
- Reading daily noise as a trend. Compare weeks and months, and annotate known changes.
- Looking at Crawl Stats in isolation. Pair it with a site crawl to find the underlying crawlability issues.
Related Guides
- Discovered – Currently Not Indexed: What It Means and How to Fix It
- 5xx Server Errors and SEO: How to Diagnose and Fix Them
- Excessive DOM Size: Why It Hurts Performance and How to Fix It
Frequently Asked Questions
How often should I check the Crawl Stats report?
Monthly is enough for most larger sites, plus extra checks after migrations, launches, hosting changes or traffic drops.
What is a good average response time in Crawl Stats?
Google does not publish a target. Look for a stable or falling trend, and investigate if response time climbs while crawl requests fall.
Why did my crawl requests suddenly drop?
Common causes include server errors, slow responses, robots.txt fetch failures, DNS problems or a firewall blocking Googlebot. Check host status and the response breakdown first.
Does the Crawl Stats report include all Google crawlers?
It covers Google crawling of your site, broken down by Googlebot type such as Smartphone, Desktop, Image, Video, AdsBot and StoreBot. Other search engines are not included.
Can I increase my crawl rate?
You cannot request more crawling directly. Fast, reliable servers, fewer low-value URLs, strong internal linking and up-to-date sitemaps encourage Google to crawl what matters.
Conclusion
The Crawl Stats report is your window into how Googlebot experiences your server. Check host status, watch for step changes, and use the breakdowns to find where crawling is wasted, and you will spot many technical problems before they cost traffic.
If your crawl data is raising more questions than answers, our technical SEO specialists can run a full crawl and log analysis as part of an SEO audit. Request a free quote.
References
- Crawl Stats report – Search Console Help
- New and improved crawl stats for your site – Google Search Central Blog
- Optimize your crawl budget – Google Search Central
- How HTTP status codes affect Google's crawlers – Google Search Central
- Crawl Budget: What Is It and Why Is It Important for SEO? – Semrush Blog
