Checking Crawl Stats and Indexing After an Algorithm Update

Your traffic falls in the same week Google confirms an update, and the obvious conclusion is that the update did it. Sometimes that is true. But updates do not stop servers from timing out, plugins from adding stray noindex tags or CDN rules from blocking Googlebot. When a technical fault lands in the same window as an update, you can spend months rewriting content that was never the problem.

This guide shows you how to use two Search Console reports, Crawl Stats and Page indexing, to rule technical causes in or out before you blame an algorithm. Google's own troubleshooting page lists technical issues, security issues, spam issues, algorithmic changes, changing search interest and site moves as separate causes of traffic drops. The aim here is to check the cheap, fixable causes first.

You will get a simple timeline method, a table of crawl patterns and what they usually mean, an indexing checklist, and a step-by-step routine you can run within a day of spotting a drop in 2026.

Key Takeaways

• A drop that lines up with an update is not proof the update caused it. Check crawling and indexing first, because those faults are quicker to confirm and fix.
• Crawl Stats shows how Googlebot is fetching your site (requests, response codes, host status). The Page indexing report shows which URLs made it into the index.
• Algorithmic drops usually show normal crawling and steady indexing with lower rankings. Technical drops usually show errors, falling crawl volume or URLs leaving the index.
• Compare the same date ranges in every report and mark the update start and end dates from the Google Search Status Dashboard.
• Small sites have less need for Crawl Stats; Google says sites under about a thousand pages usually do not need it. Page indexing and URL Inspection still apply.

Why technical problems hide behind update dates

Google publishes core and spam updates on the Google Search Status Dashboard, and most SEO tools add markers for them to their charts. That is useful, but it creates a bias: every dip near a marker looks like an algorithm effect. Meanwhile, the things that really break sites (a server migration, a security plugin that blocks unfamiliar user agents, a theme update that changes canonical tags) happen all the time and have nothing to do with Google's calendar.

Google's troubleshooting documentation describes the technical category plainly:

“Technical issues are errors that can prevent Google from crawling, indexing, or serving your pages”

— Google Search Central, Debugging drops in Google Search traffic

If Google cannot crawl or index a page, no amount of content improvement will help it rank. So the first question after any drop is simple: can Google still reach, render and index the pages that lost traffic?

What the Crawl Stats report tells you

The Crawl Stats report sits under Settings in Search Console. Google describes it as showing statistics about Google's crawling history on your website, and adds that it is aimed at advanced users. For a full walkthrough of each chart, see our guide to the Search Console Crawl Stats report. After an update, focus on four areas:

• Total crawl requests: a sudden, sustained fall can mean Googlebot is backing off because the server is slow or returning errors.
• Host status: Google checks robots.txt fetching, DNS resolution and server connectivity. Any red status in the drop window is a strong technical lead.
• Crawl responses: a rising share of 5xx, 404 or redirect responses points to server or URL problems rather than ranking changes.
• Average response time: a big jump often comes before a fall in crawl volume.

Google also notes that if your site has fewer than a thousand pages you should not need this report. For small sites, URL Inspection and the Page indexing report usually answer the same questions faster.

Crawl patterns and what they usually mean

What you see in the drop window	Likely meaning	Next check
Crawl requests steady, responses mostly 200, host status green	Crawling is healthy; the drop is more likely ranking or demand related	Move to query and page analysis
Spike in 5xx server errors	Server overload, hosting fault or firewall problem	Server logs, hosting status, uptime monitor
Host status shows robots.txt fetch failures	Google may pause crawling while robots.txt is unreachable	Fetch robots.txt yourself and check server/CDN rules
DNS resolution problems	Domain or DNS provider issue	DNS provider logs and record changes
Large rise in 404 or 301 responses	URLs changed, deleted or redirected	Recent deployments, redirect maps
Response time jumps, then crawl volume falls	Googlebot reducing load on a slow server	Performance monitoring, hosting resources
Crawl volume falls only for one file type or purpose	Possible blocking of resources (for example JavaScript or CSS)	robots.txt rules, URL Inspection rendering

None of these patterns proves a single cause. They tell you where to look next. If you want raw detail, server log file analysis shows every Googlebot request rather than the sample Search Console summarises.

What the Page indexing report adds

Crawling is only half the story. The Page indexing report shows how many of your URLs Google has crawled and indexed, and why the rest are not indexed. Our Page indexing report guide explains every status. For update analysis, watch for:

• A sharp fall in the indexed count that matches the traffic drop.
• A jump in "Excluded by 'noindex' tag", "Blocked by robots.txt" or "Page with redirect" for important templates.
• Many key pages moving to "Duplicate without user-selected canonical" or "Duplicate, Google chose different canonical than user".
• Growth in "Crawled - currently not indexed" for sections that used to be indexed.

Keep perspective. Google's help page states:

“You should not expect all URLs on your site to be indexed, only the canonical pages, as described above.”

— Search Console Help, Page indexing report

A steady number of excluded URLs is normal. What matters is change: important pages that were indexed before the drop and are not indexed now.

Step-by-step: rule out technical causes in a day

1. Mark the dates. Note the exact start and end of the update from the Search Status Dashboard and the first day your traffic changed.
2. Check the Performance report shape. A cliff that drops to near zero for whole sections suggests a technical block. A gradual slide across many queries is more typical of ranking change. Our guide on diagnosing a core update drop in Search Console covers this in depth.
3. Open Crawl Stats. Look at host status, response codes and response time for the two weeks before and after the drop.
4. Open Page indexing. Compare indexed and not-indexed counts and check which reasons grew.
5. Inspect your top losing URLs. Use URL Inspection on five to ten pages that lost the most clicks. Confirm they are indexed, that the Google-selected canonical matches yours and that the rendered page looks complete.
6. Review your change log. List deployments, plugin updates, CDN or firewall changes, hosting moves and robots.txt edits in the same window.
7. Rule out tracking. If Search Console clicks are steady but analytics sessions fell, the problem may be measurement. See tracking errors versus real ranking drops.
8. Decide. If crawling and indexing are healthy, treat the drop as a likely ranking change and move on to content and query analysis. If not, fix the technical fault first and monitor recovery.

Technical check versus algorithmic signal

Signal	Points to a technical cause	Points to an algorithmic cause
Impressions	Fall sharply for whole sections or templates	Fall gradually across many queries
Average position	Pages vanish rather than slip	Pages slip several positions but stay visible
Indexed pages	Indexed count falls	Indexed count stable
Crawl responses	More errors, redirects or timeouts	Mostly 200 responses, as before
Timing	Starts on the day of a deployment or server change	Builds during the update rollout window
Scope	Concentrated in one directory, template or subdomain	Spread across content types, often site-wide

Real cases can show both. A site may have a crawl problem on one template and a genuine ranking loss elsewhere. Splitting the analysis by template or directory, as covered in our guide to page-level versus site-wide ranking drops, keeps the two apart.

When an update and a site change coincide

The hardest case is a planned change, such as a redesign, a CMS switch or a domain move, that happens during an update. Google's documentation on site moves warns that rankings can fluctuate while it recrawls and reindexes. In that situation, check redirects and indexing for the moved URLs first, then compare them against pages that did not move. We cover the full method in migration problem or algorithm update?.

Common mistakes

• Reading Crawl Stats in isolation. A lower crawl rate on its own does not mean a penalty. Google may simply be crawling less because fewer pages are changing.
• Panicking about excluded URLs. Many exclusions (redirects, canonicalised duplicates, parameter URLs) are expected and harmless.
• Using different date ranges in different reports. Line them up, or you will see patterns that are not there.
• Ignoring the change log. Most technical drops trace back to a change someone made. Ask developers and hosting providers before guessing.
• Rewriting content before checking indexing. If a page is not indexed, editing its copy will not bring it back.

Related Guides

• Cleaning Up User-Generated Spam After a Spam Update
• Recovering When a Hacked Site Gets Caught in a Spam Update
• Cleaning Up Doorway-Style Location Pages After a Spam Update

Frequently Asked Questions

Does a drop in crawl requests mean my site was hit by an update?

Not necessarily. Crawl volume changes for many reasons, including server speed, site size and how often content changes. Treat it as a clue and check response codes, host status and indexing before drawing conclusions.

Do core updates change how often Google crawls my site?

Google has not documented a direct link between core updates and crawl rate. If crawling changes at the same time as an update, investigate technical causes such as server errors or blocked resources first.

How far back should I compare data?

Use matching windows, such as two to four weeks before the drop and the same length after. Google also recommends waiting at least a week after a core update completes before analysing results.

My site is small. Do I need the Crawl Stats report?

Google says sites with fewer than a thousand pages should not need it. URL Inspection and the Page indexing report are usually enough to confirm that important pages are crawlable and indexed.

What if crawling and indexing look normal?

Then the drop is more likely ranking or demand related. Move on to query-level analysis, compare winners and losers in the results, and review content quality.

Conclusion

Checking crawl and indexing data takes an hour or two and can save months of misdirected work. If both reports look healthy, you can analyse the update with confidence. If they do not, you have found a fix that is fully in your control. For the wider picture, start from our step-by-step ranking drop diagnosis. If you would rather have specialists run the audit, our SEO services team can review your site, or you can request a free quote.

References

1. Debugging drops in Google Search traffic (Google Search Central)
2. Crawl Stats report (Search Console Help)
3. Page indexing report (Search Console Help)
4. Google Search's core updates (Google Search Central)
5. Site moves with URL changes (Google Search Central)
6. Google Search Status Dashboard: Ranking incident history