When you search on Google, you get results in a fraction of a second. So does Google go out and read the whole internet every time you hit Enter? It doesn't, and understanding what it does instead is one of the most useful things you can learn as a beginner.
In this lesson, I will walk you through the three stages Google uses to go from billions of web pages to the handful of results you see: crawling, indexing and serving. By the end, you will be able to explain how Google finds a new page, why some pages never show up in search, and which simple checks tell you whether Google knows about your site.
If you are new to the course, the first lesson on what SEO is gives you the basic vocabulary we use here.
What are the three stages of Google Search?
Google explains its own process in a document called "In-depth guide to how Google Search works." It describes three stages, and it makes an important point: not every page makes it through every stage.
| Stage | What happens | Library analogy |
|---|---|---|
| 1. Crawling | Google's software finds pages and downloads their text, images and videos. | A librarian walks around collecting every new book they can find. |
| 2. Indexing | Google analyzes each page and stores information about it in a huge database called the index. | The librarian reads each book, notes what it's about and files it in the catalog. |
| 3. Serving | When someone searches, Google picks the most relevant results from the index and shows them. | A reader asks a question and the librarian recommends the best books from the catalog. |
The key thing to understand is that when you search, Google isn't searching the live web. It is searching its index, which is its own stored copy of what it has found. That's why it can answer so fast. It also means that if your page isn't in the index, it simply can't appear in the results.
Stage 1: how does Google find and crawl pages?
Discovery: there is no master list of websites
Google says there is no central registry of all web pages. Nobody hands Google a list. So it has to constantly look for new and updated pages. This first step is called URL discovery (a URL is simply a web address, like example.com/cakes).
According to Google, it finds new pages in three main ways:
- By revisiting pages it already knows. For example, it goes back to a blog's home page and notices a new post listed there.
- By following links. When one known page links to a new page, Google can discover it. This is the main way Google finds pages, and it's why links matter so much. Internal links on your own site help too, which we cover in the lesson on internal linking.
- By reading sitemaps. A sitemap is a file that lists the URLs on your site that you want Google to know about. Google says sitemaps can help but aren't required. The lesson on XML sitemaps shows how to create one.
Crawling: Googlebot visits the page
Once Google knows a URL exists, it may visit it. The program that does this is called Googlebot, Google's main crawler (a crawler, sometimes called a spider or bot, is software that automatically visits web pages and downloads them).
A few facts from Google's documentation are worth knowing:
- Googlebot decides what to crawl. It uses an automatic process to choose which sites to crawl, how often and how many pages to fetch from each.
- It tries not to overload your site. Google says Googlebot shouldn't access your site more than once every few seconds on average, and it slows down if your server returns errors.
- It mostly crawls as a phone. Google primarily indexes the mobile version of sites, so most crawl requests come from Googlebot Smartphone. This is why mobile-friendly design matters, as you will see in the lesson on mobile-first indexing.
- It doesn't crawl everything it finds. Some pages are blocked by the site owner, and some sit behind a login.
Site owners can tell crawlers which parts of a site not to visit using a small file called robots.txt. You will learn how it works in the lesson on robots.txt. If you want more detail on the crawler itself, my post on how Googlebot works goes deeper.
Rendering: seeing the page like a browser does
Many modern websites use JavaScript (a programming language that runs in your browser) to load content after the page opens. If Google only read the raw code, it might miss that content. So, as part of crawling, Google renders the page. Google says it uses a recent version of Chrome to do this and runs any JavaScript it finds, much like your own browser does.
Rendering usually works fine, but heavy reliance on JavaScript can cause problems, such as content that never appears to Google. The lesson on JavaScript SEO explains what to watch for.
Stage 2: what happens when Google indexes a page?
After a page is crawled, Google tries to understand what it's about. This is indexing. Google says it analyzes the text, key tags and attributes such as the title element and image alt text, plus images and videos.
Choosing one version among duplicates
The web is full of near-identical pages. A product might be reachable at three slightly different URLs, or a page might have a printer-friendly copy. Google groups similar pages into a cluster and picks one to represent the group. This chosen page is called the canonical, and it's the version that may be shown in search results. The other versions may still be shown in some situations, for example to a user on a mobile device.
You can give Google hints about which version you prefer. The lesson on canonical tags explains how.
Storing signals about the page
Google also records details about the canonical page, such as its language, the country the content is aimed at and how usable it is. It stores all of this in the Google index, which Google describes as a large database. These stored details are used later, when deciding which results to show.
Indexing is not guaranteed
This surprises many site owners. Google says plainly that not every page it crawls will be indexed. Common reasons it gives include:
- The content is low quality.
- A robots meta rule (such as
noindex) tells Google not to index the page. - The website's design makes indexing difficult.
The lesson on indexing in chapter 5 covers how to diagnose these issues. If you already see the "Crawled – currently not indexed" message in Search Console, my post on fixing "crawled, currently not indexed" will help.
Stage 3: how does Google serve search results?
When someone types a search, Google looks through its index for matching pages and returns the ones it believes are the highest quality and most relevant. Google says relevance depends on hundreds of factors, which can include the user's location, language and device.
That's why two people can search for the same thing and see different results. Google gives a nice example: someone searching "bicycle repair shops" will probably see local results, while someone searching "modern bicycle" is more likely to see image results. The type of results, not just their order, changes with the search.
How Google decides the order is a big topic, so it gets its own lesson: how Google ranks pages.
Google says it doesn't accept payment to crawl a site more often or rank it higher. Anyone who tells you otherwise is wrong.
Why might a page not show up in Google?
Because each stage is a filter, a page can drop out at any point. Here is a simple way to think about it, based on the reasons Google lists:
| Where it fails | Typical reasons | Where to learn more |
|---|---|---|
| Not crawled | Server problems, network issues, robots.txt blocking the page, no links pointing to it | robots.txt lesson |
| Crawled but not indexed | Low-quality or duplicate content, a noindex rule, a hard-to-process design | Indexing lesson |
| Indexed but not shown | Content isn't relevant to what people search, quality is low, or robots rules prevent it being served | Ranking lesson |
Google also notes that a page can be indexed but simply never appear for the searches you hoped for. Being in the index is the entry ticket, not a promise of traffic. My longer post on how search engines work adds more examples of each failure point.
Try it: check whether Google has found your pages
Here's a short, practical check you can do today. It shows you how Google currently sees your site.
- Do a quick site search. In Google, type
site:yourdomain.com(replace with your own domain). This shows a sample of pages Google has indexed from your site. It's a rough check, not an exact count. - Look for something missing. Think of your most important pages, such as your main services or best products. Search
site:yourdomain.complus a word from that page's title. If it doesn't appear, make a note. - Set up Google Search Console. It's a free tool from Google. If you haven't set it up yet, the lesson on Google Search Console walks you through it.
- Inspect one important URL. In Search Console, paste a page address into the URL Inspection bar at the top. It tells you whether the page is indexed and, if not, why. My post on the URL Inspection tool explains every line of the report.
- Open the Pages report. Under "Indexing," the Pages report lists indexed and not-indexed pages with reasons. My guide to the page indexing report explains the common messages.
- Check that your key pages are linked. Make sure each important page is linked from at least one other page on your site, such as your menu or a related article. Pages with no links pointing to them are hard for Google to find.
Don't worry if some pages aren't indexed. That's normal for things like tag pages or duplicates. What matters is that the pages you care about are in the index.
Questions beginners ask about how search engines work
Do I need to submit my site to Google for it to be found?
Not usually. Google finds most pages by following links. Submitting a sitemap through Search Console can help Google discover your pages, especially on a new site with few links, but Google says it isn't required.
How often does Google crawl my site?
There's no fixed schedule. Googlebot decides automatically how often to visit each site and page. Pages that change often and are well linked tend to be visited more often. For very large sites, my post on crawl budget explains the limits.
Can I pay Google to index or rank my site faster?
No. Google says it doesn't accept payment to crawl a site more often or rank it higher. Paid ads are a separate product and don't affect organic rankings.
Does Bing work the same way?
In broad terms, yes. Most search engines crawl, index and then rank pages. The details and tools differ, but a site that is easy for Google to crawl and understand is usually easy for other search engines too.
From crawl to results: what to remember, and the next lesson
Google finds pages mainly by following links, crawls them with Googlebot, renders them like a browser, and then decides whether to store them in its index. Only indexed pages can appear in results, and even then Google picks what to show based on relevance, quality and the searcher's situation. When a page is missing from Google, ask which of the three stages it failed at.
In the next lesson, how Google ranks pages, we look at the systems Google uses to decide which indexed pages come first. If you find indexing problems on your own site and want a hand, our SEO team is happy to take a look.