If Google cannot reach a page, nothing else you do for that page matters. The best copy, the cleverest internal links and the most carefully tuned title tag are all invisible until a crawler can request the URL, receive a sensible response and follow links onward to the rest of your site. That is what crawlability means: the ability of search engine bots to discover and fetch your pages.
Crawlability problems are sneaky because they rarely announce themselves. Rankings slide, new pages take weeks to show up, or a product category quietly disappears from search. By the time someone notices, the cause might be a single line in robots.txt, a JavaScript menu that hides links, or a server that buckles every time Googlebot visits.
This guide walks through the crawlability issues we see most often in technical audits, how to diagnose each one with free tools, and a step-by-step process to fix them in 2026. It focuses on the crawling stage specifically; indexing and ranking problems are covered in other guides we link to along the way.
Key Takeaways
- Crawlability is about access: can a bot find the URL, request it and get a usable response?
- Robots.txt controls crawling, not indexing. Blocking a page does not guarantee it stays out of Google.
- Google reliably follows only
<a>elements with anhrefattribute, so JavaScript-only navigation is a common blind spot. - Persistent 5xx errors slow crawling and can eventually remove URLs from the index.
- Search Console's Crawl Stats and Page indexing reports, plus a desktop crawler, will surface most issues.
Crawlability vs Indexability: What Is the Difference?
People often use the two words interchangeably, but they describe different stages. Crawling is the act of fetching a URL. Indexing is Google deciding to store and potentially show that URL. A page can be crawlable but not indexed (for example, it has a noindex tag or Google judges it low value), and a page can occasionally be indexed without being crawled (when it is blocked by robots.txt but linked from elsewhere).
If you want the bigger picture of how these stages fit together, our explainer on how Googlebot works covers the full pipeline. In this article we stay focused on the first gate: access.
| Stage | Question it answers | Typical blockers | Where to check |
|---|---|---|---|
| Discovery | Does Google know the URL exists? | Orphan pages, missing sitemap, JS-only links | Sitemaps report, crawler link graph |
| Crawling | Can Google fetch it? | Robots.txt disallow, 5xx errors, timeouts, login walls | Crawl Stats report, URL Inspection |
| Rendering | Can Google see the content? | Blocked JS/CSS, client-side-only content | URL Inspection > View crawled page |
| Indexing | Will Google store it? | noindex, canonical to another URL, thin or duplicate content | Page indexing report |
The Most Common Crawlability Issues
1. Robots.txt rules that block too much
A robots.txt file is a blunt instrument. One stray Disallow: / left over from a staging environment can shut out an entire site, and wildcard rules written to block filter parameters sometimes catch real category pages too. Google is clear about what the file is for:
“it is not a mechanism for keeping a web page out of Google”
— Google Search Central, Introduction to robots.txt
The same documentation explains that a blocked URL can still appear in results, just without a description. So robots.txt is the wrong tool for hiding pages and a dangerous tool if misconfigured. For the difference between the file and on-page directives, see our guide to robots.txt vs meta robots. Staging environments deserve special care: their robots.txt rules should never be copied to the live site on launch day.
2. Links that crawlers cannot follow
Google's link documentation says that, generally, it can only crawl a link if it is an <a> HTML element with an href attribute. Menus built from <span onclick>, buttons that change the URL with JavaScript, and "load more" pagination without real links all reduce what Googlebot can discover. Our JavaScript SEO guide goes deeper on rendering, but the quick fix is simple: every navigational element should resolve to a real anchor with a real URL.
3. Orphan pages and weak internal linking
A page with no internal links pointing to it relies entirely on your sitemap or external links to be found. Even when Google knows the URL, an orphan sends a signal that you don't consider it important. Our guide to internal linking covers the fix in detail.
4. Server errors and timeouts
Google's crawlers back off when a server struggles. According to Google's documentation on HTTP status codes, 5xx and 429 responses make crawlers temporarily slow down, and URLs that persistently return server errors are eventually removed from the index. If your hosting falls over under load, you are effectively asking Google to crawl less. We cover diagnosis in fixing 5xx server errors.
5. Redirect chains and loops
Every hop in a redirect chain is another request. Long chains waste crawl activity and loops can trap bots entirely. Flatten chains so each old URL points directly at its final destination; our guide to redirect chains shows how.
6. Infinite URL spaces
Faceted filters, calendar widgets, session IDs and sort parameters can generate near-endless URL variations. Crawlers spend their time on low-value permutations while important pages wait. This is closely tied to index bloat and, on large sites, to crawl budget.
7. Access restrictions
Pages behind logins, IP allowlists, geo-blocks or aggressive bot protection may return 401, 403 or challenge pages to Googlebot. Firewalls and CDN security rules are frequent culprits after a hosting change; check that verified Googlebot requests are allowed through.
8. Broken or missing sitemaps
Google's sitemap documentation says small sites (about 500 pages or fewer) that are comprehensively linked internally may not need a sitemap at all. Larger, newer or media-heavy sites benefit from one, but a sitemap full of redirected, noindexed or 404 URLs sends mixed signals. Keep it clean: list only canonical, indexable 200 URLs.
How to Find Crawlability Issues
You don't need expensive software to start. Most problems surface with a combination of Google's own data and a desktop crawler.
| Tool | What it reveals | Cost |
|---|---|---|
| Search Console Crawl Stats report | Request volume, response codes, host status (robots.txt fetching, DNS resolution, server connectivity) | Free |
| Search Console Page indexing report | "Blocked by robots.txt", "Server error (5xx)", "Discovered – currently not indexed" | Free |
| URL Inspection tool | Live fetch result, rendered HTML, robots.txt status for one URL | Free |
| Screaming Frog, Sitebulb or similar | Full link graph, orphan pages, redirect chains, blocked resources | Free tier / paid |
| Server log files | Exactly which URLs Googlebot requested and what it received | Free if you have access |
Google notes that the Crawl Stats report is aimed at advanced users and that sites with fewer than a thousand pages should not normally need it. For bigger sites, it is the fastest way to see host problems. We explain each chart in our Crawl Stats report walkthrough. If many URLs sit in "Discovered – currently not indexed", Google knows about them but hasn't fetched them yet, which often points to crawl capacity or quality concerns; see our guide to that status.
Step-by-Step: Fixing Crawlability Issues
- Test robots.txt first. Open
yourdomain.com/robots.txtand read every rule. Check key templates (home, category, product, article) with the URL Inspection tool to confirm they are not blocked. - Crawl the site like a bot. Run a desktop crawler from the home page with JavaScript rendering off, then on. Compare the URL counts. A big gap means important links depend on JavaScript.
- Cross-check against your sitemap. URLs in the sitemap but not found by the crawl are orphans. URLs found by the crawl but missing from the sitemap may be low-value pages you should consolidate.
- Review status codes. Export every non-200 URL. Fix internal links that point at 404s or redirects so they go straight to the live 200 URL.
- Check server health. In Crawl Stats, look for spikes in server errors or rising average response time. Talk to your host or developer about capacity, caching and bot rules.
- Tame parameter URLs. Use canonical tags, consistent internal linking and, where truly necessary, robots.txt rules to stop crawlers wandering through endless filter combinations.
- Request re-crawling of fixed pages. Resubmit the sitemap and use URL Inspection for a handful of priority pages. Then watch the Page indexing report over the following weeks.
Common Mistakes When Fixing Crawlability
- Using robots.txt to remove pages from Google. If a page is blocked, Google can't see a
noindextag on it. Allow crawling and usenoindexinstead. - Blocking CSS and JavaScript files. Google needs these to render the page as users see it.
- Fixing the sitemap but not the links. A sitemap is a hint. Internal links are the stronger, more durable signal.
- Ignoring the CDN or firewall. Bot-protection rules that challenge unknown user agents can block Googlebot without anyone on the marketing team noticing.
- Chasing tool scores. Audit tools flag hundreds of "issues". Prioritise the ones that block important templates over cosmetic warnings.
Crawlability Checklist by Site Type
Different sites break in different ways. Use this as a starting point when planning an audit.
- Small business brochure sites: robots.txt, contact and service pages linked from the menu, no leftover staging blocks.
- Ecommerce stores: faceted navigation, out-of-stock handling, pagination links, product variant URLs.
- Publishers and blogs: archive pagination, tag pages, author pages, AMP or legacy URL redirects.
- JavaScript apps: server-side or static rendering for key content, real anchor links, no hash-based routing for indexable views.
Get Your Site Crawled Properly
Crawlability fixes are often small changes with outsized impact, but finding them across thousands of URLs takes time and the right tools. If you'd like experienced eyes on your site, our technical SEO specialists can run a full crawl and log review as part of our SEO services. Request a free quote and we'll tell you what's holding Google back.
References
- Introduction to robots.txt – Google Search Central
- Link best practices for Google
- How HTTP status codes affect Google's crawlers
- Crawl Stats report – Search Console Help
- Learn about sitemaps – Google Search Central
- 15 Crawlability Problems & How to Fix Them – Semrush
