Before a page can rank, Google has to fetch it. That job belongs to Googlebot, the crawler that requests billions of URLs, follows links, renders JavaScript and hands what it finds to Google's indexing systems. If Googlebot cannot reach a page, or reaches only a broken version of it, nothing else in your SEO strategy can compensate.
Yet many site owners treat Googlebot as a black box. They do not know which version visits, how much of each file it reads, why it slows down, or how to tell a real Googlebot from a scraper pretending to be one.
This guide explains how Googlebot works in 2026, step by step, using Google's own documentation, and shows you how to monitor and influence its behaviour on your site.
Key Takeaways
- Googlebot is the name for two crawlers, Smartphone and Desktop; most crawling uses the smartphone crawler because of mobile-first indexing.
- It discovers URLs from links, sitemaps and previously known pages, but does not crawl every URL it finds.
- For Google Search it fetches the first 2MB of a supported file (64MB for PDFs).
- Pages are rendered with an evergreen version of Chromium, so JavaScript content can be seen, but rendering may be queued.
- Crawl speed adapts to your server's health; fast, error-free responses allow more crawling.
- Verify real Googlebot with reverse DNS or Google's published IP ranges, because the user agent is easy to fake.
What Is Googlebot?
Googlebot is Google's main web crawler for Google Search. According to Google's Googlebot documentation, it is a generic name for two crawler types:
| Crawler | Simulates | Share of crawling | robots.txt token |
|---|---|---|---|
| Googlebot Smartphone | A user on a mobile device | Most requests (mobile-first indexing) | Googlebot |
| Googlebot Desktop | A user on a desktop computer | A smaller share | Googlebot |
Because both obey the same product token, you cannot use robots.txt to block one type and allow the other. Google also runs many other crawlers and fetchers, such as Googlebot-Image, Googlebot-Video, Storebot-Google and AdsBot, listed in its crawler overview. Since the smartphone crawler does most of the work, your mobile pages are what Google mainly sees; content parity between mobile and desktop is essential.
How Googlebot Works, Step by Step
- URL discovery. Google finds URLs by following links from pages it already knows, reading XML sitemaps and revisiting previously crawled pages.
- Scheduling. An algorithmic process decides which URLs to crawl, how often and how many to fetch from each site.
- robots.txt check. Before fetching, Googlebot checks your robots.txt file to see whether the URL is allowed.
- Fetching. Googlebot requests the URL over HTTP/1.1 or HTTP/2 and reads the response code, headers and content.
- Rendering. Google's Web Rendering Service runs the page's JavaScript in a recent Chromium to see the final content.
- Link extraction. New links from the rendered page are added to the crawl queue.
- Handoff to indexing. The content is analysed, duplicates are clustered and a canonical is selected for the index.
Our companion article on how search engines work covers the indexing and ranking stages that follow.
“However, Googlebot doesn't crawl all the pages it discovered.”
— Google Search Central, In-depth guide to how Google Search works
Discovery: links and sitemaps
Links remain Googlebot's main road map, which is why pages with no internal links (orphan pages) are often crawled rarely or never. XML sitemaps help Google find URLs, especially on large or new sites, and the lastmod value helps it spot updated pages. See our XML sitemaps guide for setup.
Fetching: what Googlebot actually downloads
“When crawling for Google Search, Googlebot crawls the first 2MB of a supported file type.”
— Google Search Central, Googlebot
Content beyond that point is not considered for indexing. Google's broader crawling infrastructure has a default limit of 15MB, but Googlebot for Search uses the smaller limit, while PDFs get 64MB. Each resource referenced in the HTML, such as CSS and JavaScript, is fetched separately. Most normal pages are far below 2MB of HTML, but bloated inline code, huge embedded data or endless markup can push important content past the cut-off.
Google's crawler documentation also notes that its crawlers support HTTP/1.1 and HTTP/2, support gzip, deflate and Brotli compression, and support HTTP caching, with ETag preferred over Last-Modified. It adds that Google egresses primarily from IP addresses in the United States, which matters if you serve different content by location.
Rendering: how Googlebot handles JavaScript
Google's JavaScript SEO basics describes three phases: crawling, rendering and indexing. Pages are queued for rendering, and a page may stay in that queue for a few seconds or longer. Google renders with an evergreen version of Chromium, so modern JavaScript usually works, but content that depends on user interactions (clicks, scrolls) or on resources blocked by robots.txt may never appear.
If your content depends on client-side JavaScript, read our JavaScript SEO guide and compare server-side vs client-side rendering.
How Often Does Googlebot Crawl?
Google's documentation says that, for most sites, Googlebot shouldn't access the site more than once every few seconds on average, although rates vary. How much it crawls is governed by two factors described in its crawl budget guide:
- Crawl capacity limit: how much Google can crawl without overloading your server. It rises when responses are fast and stable and falls when latency increases or 5xx errors appear.
- Crawl demand: how much Google wants to crawl, based on popularity, how often content changes and how many URLs it perceives the site to have.
For most small and medium sites, crawl budget is not a concern. Large sites with many URLs, faceted navigation or frequent updates should read our crawl budget guide and keep URL parameters under control. Server performance matters too; see how web hosting affects SEO.
How to Control Googlebot
| Goal | Method | Notes |
|---|---|---|
| Stop crawling of a section | robots.txt Disallow | Blocks crawling, not indexing of linked URLs |
| Keep a page out of the index | noindex meta tag or X-Robots-Tag | The page must be crawlable for Google to see the tag; see our noindex guide |
| Protect private content | Password protection / login | The only reliable way to keep content private |
| Signal updated content | Sitemaps with accurate lastmod | Helps scheduling; not a guarantee |
| Reduce crawl load in an emergency | Return 500, 503 or 429 | Only for a short time, per Google |
| Request a recrawl of a URL | URL Inspection "Request indexing" | Useful for single, important URLs |
Our explainer on robots.txt vs meta robots covers the difference in detail. Here is a simple robots.txt that blocks internal search for all crawlers and declares a sitemap:
User-agent: *
Disallow: /search
Sitemap: https://www.example.com/sitemap.xmlIf Googlebot is overloading your server, Google's guidance on reducing crawl rate says you can return a 500, 503 or 429 status code instead of 200 for a short period, but not for longer than one or two days, because persistent errors can lead to URLs being dropped.
How to Verify Real Googlebot Visits
Anyone can put "Googlebot" in a user agent string. Google's verification guide describes two methods:
- Run a reverse DNS lookup on the requesting IP address (for example with the
hostcommand). - Check that the hostname ends in
googlebot.com,google.comorgoogleusercontent.com. - Run a forward DNS lookup on that hostname.
- Confirm it resolves back to the original IP address.
$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1For automated checks, match IPs against Google's published JSON IP range files. This matters for log file analysis, where fake bots can distort your data.
How to Monitor Googlebot Activity
- Crawl Stats report (Search Console, Settings): total requests, average response time, responses by code, file type, crawler type and host status.
- URL Inspection tool: last crawl date, crawler used, and a live test with the rendered HTML and screenshot.
- Page indexing report: URLs discovered or crawled but not indexed, and errors such as soft 404s.
- Server logs: the raw truth of which URLs Googlebot requests and how often.
Common Mistakes
- Blocking CSS and JavaScript in robots.txt. Googlebot then renders a broken page.
- Using robots.txt to deindex pages. Blocked URLs can still be indexed from links; use noindex instead.
- Different content for mobile. Missing content on the mobile version is missing for most of Google's crawling.
- Firewalls blocking Google. Security tools that challenge or rate-limit verified Googlebot cause crawl errors.
- Trusting the user agent alone. Always verify before whitelisting or analysing "Googlebot" traffic.
- Endless crawl traps. Calendars, filters and session IDs can generate infinite URLs.
Related Guides
- The Noindex Tag: When and How to Use It
- How to Test and Fix Structured Data
- Server-Side Rendering vs Client-Side Rendering for SEO
- How Search Engines Work: Crawling, Indexing and Ranking
Frequently Asked Questions
What is Googlebot?
Googlebot is the web crawler Google uses to fetch pages for Google Search. It comes in two types, Googlebot Smartphone and Googlebot Desktop, which share the same robots.txt token.
How often does Googlebot crawl my site?
It varies by site. Google says that for most sites it should not access the site more than once every few seconds on average, and crawl frequency depends on server health, popularity and how often content changes.
Can Googlebot read JavaScript?
Yes. Google renders pages using an evergreen version of Chromium. Rendering can be delayed in a queue, and content that requires user interaction or blocked resources may not be seen.
How do I know if a visit is really from Googlebot?
Run a reverse DNS lookup on the IP, confirm the hostname ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup to confirm it matches the original IP. Or compare against Google's published IP ranges.
How much of a page does Googlebot read?
For Google Search, Googlebot fetches the first 2MB of a supported file type, and 64MB for PDFs. Each referenced resource such as CSS or JavaScript is fetched separately.
Can I make Googlebot crawl my site faster?
You cannot pay for more crawling, but fast, error-free server responses, clean internal linking, accurate sitemaps and fewer low-value URLs all help Google crawl more of what matters.
Conclusion
Googlebot is not mysterious once you know its routine: discover URLs, check robots.txt, fetch up to its size limit, render, extract links and pass the results on for indexing. Make that routine easy with fast servers, crawlable links, accurate sitemaps and render-friendly pages, and verify what it is doing with Crawl Stats and logs.
Want to see exactly how Google crawls your site? Our technical SEO service includes crawl and log analysis. Get a free quote.
References
- Google Search Central: Googlebot
- Google Crawling Infrastructure: Overview of Google crawlers and fetchers
- Google Crawling Infrastructure: Verifying Google crawlers and fetchers
- Google Search Central: Understand JavaScript SEO basics
- Google Search Central: Crawl budget management for large sites
- Google Search Central: Reduce the Google crawl rate
- Google Search Central: In-depth guide to how Google Search works
- Semrush: What Is Googlebot? How Google's Web Crawler Works
