Google Indexing: Get Pages Indexed or Keep Them Out

You published a new page a week ago, you search for it on Google, and nothing shows up. Or you open Search Console and see a long list of pages marked “not indexed.” Is something broken? Did Google reject your site? These are the most common worries I hear from people who are new to technical SEO.

In this lesson, I will explain what indexing really is, how to check whether a page is in Google’s index, what the main statuses in Search Console mean, and how to keep the pages you don’t want in search out of it. By the end, you will be able to look at any page on your site and say, with confidence, whether it is indexed, why or why not, and what (if anything) to do about it.

What does “indexed” actually mean?

Back in the lesson on how search engines work, we saw that Google does three jobs: it crawls pages, it indexes them, and it serves them in results. Indexing is the middle step. After Googlebot (Google’s crawler) downloads a page, Google tries to understand it and then decides whether to store it in its index, which is a huge database of pages that can appear in search results.

Think of a library. Crawling is the librarian picking up a new book. Indexing is the librarian deciding to catalog it and put it on a shelf. Only books on the shelf can be lent out. In the same way, only indexed pages can show up when someone searches.

Two things are important to understand early:

Being crawled does not mean being indexed. Google can read a page and still choose not to store it.
Being indexed does not mean ranking well. An indexed page is eligible to appear. Where it appears depends on everything else you learn in this course, such as helpful content and links.

Google also says plainly that it does not expect every URL on your site to be indexed. Its Page indexing report help page tells site owners not to aim for 100% coverage. Duplicates, redirected URLs and pages you deliberately keep out are normal. Your goal is simple: the important, original version of each useful page should be indexed.

How do you check if a page is indexed?

There are two tools I suggest beginners learn. Both are inside Google Search Console, which is free.

The URL Inspection tool, for one page

Paste any URL from your site into the search bar at the top of Search Console. The tool tells you whether the URL is on Google, when it was last crawled, which canonical URL Google selected, and whether anything blocked it. You can also click “Test live URL” to see how Googlebot sees the page right now.

One thing confuses people here. The main result shows the indexed version, meaning what Google saw the last time it crawled. The live test shows the page as it is today. If you just fixed something, the two can disagree for a while, and that is expected. I go deeper into every field in my guide to the URL Inspection tool.

The Page indexing report, for the whole site

Under “Indexing” then “Pages,” Search Console shows how many of your known URLs are indexed and how many are not, grouped by reason. This is where you spot patterns, for example “200 product filter URLs are excluded as duplicates.” My separate post on the Search Console Page indexing report explains every status in detail.

You may also have heard of typing site:yourdomain.com into Google. It gives a rough idea, but the counts are estimates and not reliable for checking a single page. For real answers, use Search Console.

What do the Page indexing report statuses mean?

The report lists many reasons why a page is not indexed. Some need action and many don’t. Here are the ones beginners see most, in plain words, based on Google’s own help page for the report.

Status	What it means	Do you need to act?
Crawled – currently not indexed	Google visited the page but decided not to index it, at least for now.	Maybe. Improve the page if it is important; there is no need to keep resubmitting it.
Discovered – currently not indexed	Google knows the URL exists but has not crawled it yet. It rescheduled the crawl.	Usually wait. If many important pages sit here, look at site quality and internal links.
URL marked ‘noindex’	The page tells Google not to index it.	Only if you actually want it indexed. Then remove the noindex.
Alternate page with proper canonical tag	This page points to another page as the main version, and Google agrees.	No. This is working as intended.
Duplicate without user-selected canonical	Google sees this as a duplicate and picked another URL as the main version.	Check which URL Google picked. Set a canonical if you prefer a different one.
Duplicate, Google chose different canonical than user	You named a main version, but Google chose another.	Review your signals. Covered in the canonical tags lesson.
Page with redirect	The URL redirects somewhere else, so it is not indexed itself.	No, as long as the redirect is intentional.
Soft 404	The page looks like an error or empty page but returns a “success” code.	Yes. Return a real 404 for missing pages, or add real content.
Blocked by robots.txt	Your robots.txt file stops Google from crawling the URL.	Only if the block is a mistake.

Two statuses deserve a little more attention, because they are the ones that worry people the most.

“Crawled – currently not indexed”

This usually means Google did not think the page was worth storing right now. Common reasons are thin content, pages that are very similar to other pages, or a page that offers nothing new compared to what is already in the index. Google’s help page notes that the page may be indexed later, and that you don’t need to resubmit it. My advice is to ask, honestly, “Is this page clearly useful and different?” If not, improve it, merge it into a stronger page, or accept that it doesn’t need to be indexed. I have a full walkthrough in how to fix Crawled – currently not indexed.

“Discovered – currently not indexed”

Here Google knows about the URL, often from a link or your XML sitemap, but hasn’t fetched it yet. Google’s explanation is that crawling it at that moment might have overloaded the site, so it pushed the crawl to later. On a small site, this usually clears up by itself. On a big site with thousands of weak URLs, it can be a sign that Google doesn’t think it is worth crawling them all. See Discovered – currently not indexed for the deeper version.

How do you get an important page indexed faster?

You can’t force Google to index a page. But you can make it easy to find, easy to crawl and clearly worth indexing. Here is how I would do it for a new or updated page.

Make sure the page can be crawled. It must not be blocked in robots.txt, and it must return a normal 200 status code.
Remove any accidental noindex. Check the page source for a robots meta tag. Website builders sometimes add noindex to new pages or to the whole site while it is in “draft” mode.
Link to it from relevant pages. Add at least one link from a related page that is already indexed, such as a category page or a popular article. Links are how Googlebot discovers most pages.
Add it to your XML sitemap. Most SEO plugins do this automatically. For many new or updated pages at once, Google recommends a sitemap over individual requests.
Run a live test in URL Inspection. Confirm Googlebot can fetch and render the page without errors.
Click “Request indexing.” This asks Google to recrawl the page. You can only do it for a limited number of URLs per day, and only if the live test says the page is indexable.
Be patient. Google says crawling can take from a few days to a few weeks, and requesting the same URL again and again won’t make it faster.

Please keep one honest point in mind. Google’s own documentation says that submitting a request does not guarantee indexing. If a page keeps getting crawled and not indexed, the problem is usually the page, not the button.

How do you keep a page out of Google?

Not every page belongs in search. Thank-you pages, internal search results, login pages, test pages and thin tag archives usually don’t help searchers. Too many of them can also lead to index bloat, where lots of low-value URLs dilute your site.

The right tool for keeping a page out of Google’s index is the noindex rule. You can add it in two ways:

A meta tag in the page’s <head>: <meta name="robots" content="noindex">
An HTTP header sent by the server: X-Robots-Tag: noindex. This is handy for PDFs and other files that have no HTML head.

If you use WordPress, Wix or a similar platform, you usually don’t edit code. Your SEO plugin or page settings have a “hide from search engines” or “noindex” option. My post on the noindex tag shows where to find it on common platforms.

The robots.txt trap

This is the single most common indexing mistake I see, so let me explain it carefully. Many people block a page in robots.txt and add noindex to it. That does not work. Google’s documentation says that for noindex to be effective, the page must not be blocked by robots.txt. If Googlebot is not allowed to crawl the page, it never sees the noindex tag.

Worse, a URL blocked by robots.txt can still be indexed if other pages link to it. It shows up in Search Console as “Indexed, though blocked by robots.txt,” usually with a poor or missing description. So remember the difference:

robots.txt controls crawling (whether Googlebot may visit).
noindex controls indexing (whether the page may appear in results).

Removing something quickly

If a page must disappear from results urgently, Search Console has a Removals tool. It hides a URL for about six months, but it is temporary. For a permanent result, you must also delete the page (so it returns a 404 or 410 status), put it behind a login, or add noindex. My guide on removing pages from Google Search walks through each case.

Try it: audit the indexing of five pages on your site

Now let’s put this into practice. Set aside about 20 minutes and do the following on your own site.

Pick five pages: your home page, one main service or product page, one recent blog post, your contact page and one page you think should not be in Google (for example a thank-you page).
Paste each URL into the URL Inspection tool in Search Console.
For each page, write down three things: is it on Google, when was it last crawled, and which canonical did Google select?
For any important page that is not indexed, note the reason shown. Match it to the table above.
For the page you want kept out, check whether it has noindex and whether robots.txt is blocking it. If both, remove the robots.txt block so Google can see the noindex.
Open the Page indexing report and look at the list of “Why pages aren’t indexed.” Click the biggest group and scan the example URLs. Are they pages you care about, or pages that are fine to leave out?

Here is an example of what your notes might look like (invented, for illustration):

Page	On Google?	Reason / note	Action
/services/roof-repair/	Yes	Canonical matches	None
/blog/gutter-cleaning-tips/	No	Discovered – currently not indexed	Link to it from two related posts; wait
/thank-you/	Yes	No noindex on the page	Add noindex in SEO plugin

If you fix something that appears as an issue in the Page indexing report, you can click “Validate fix.” Google’s help page says validation usually takes up to about two weeks.

Questions beginners ask about indexing

How long does it take Google to index a new page?

There is no fixed time. Google says crawling can take from a few days to a few weeks. Pages on well-linked, regularly updated sites tend to be picked up faster than pages on new sites with few links.

Should I request indexing for every new blog post?

You can, but you don’t have to. For a few important pages it is fine. For many pages, a sitemap is the better route, and good internal links do most of the work anyway. Requesting the same URL many times does not help.

Is it bad to have lots of “not indexed” pages?

Not by itself. Redirects, duplicates and noindexed pages are supposed to be not indexed. It only matters when pages you want in search are in that list, or when your site creates huge numbers of low-value URLs.

Can I use robots.txt to remove a page from Google?

No. robots.txt stops crawling, not indexing, and it prevents Google from seeing a noindex tag. Use noindex, delete the page, or put it behind a login instead.

My page was indexed and then disappeared. Why?

Check URL Inspection first. Common causes are an accidental noindex, a new robots.txt block, a redirect, a canonical pointing elsewhere, or a server error. If none apply, Google may have decided the page is not useful enough, which is a content question.

Indexing in one paragraph, and where we go next

In brief: indexing is Google’s decision to store a page so it can appear in results. Use URL Inspection to check single pages and the Page indexing report to spot patterns. Help important pages get indexed with crawlable pages, internal links and a sitemap. Keep unwanted pages out with noindex, not robots.txt. And don’t chase 100%, because many URLs are meant to stay out.

Many “not indexed” statuses are really about duplicates, so the next lesson is canonical tags explained, where you will learn how to tell Google which version of a page is the main one. If you would rather have someone check your site’s indexing for you, our team is happy to help; just ask for a free quote.

Getting Pages Indexed (and Keeping Some Out)