Robots.txt for Beginners: Control What Google Crawls

Somebody told you to "check your robots.txt", or you saw a warning about it in Search Console, and now you are worried. What is this file? Could one wrong line really hide your whole site from Google? And should you be using it to block AI bots?

The short answers: robots.txt is a small text file with instructions for crawlers, and yes, one wrong line can cause real damage. The good news is that the rules are simple once you see them laid out. After this lesson, you will be able to read any robots.txt file, write basic rules, avoid the classic mistakes, and decide how to handle AI crawlers.

What is a robots.txt file?

A robots.txt file is a plain text file that sits at the root of your website, for example https://www.example.com/robots.txt. It tells crawlers (the automated programs, also called bots, that visit pages for search engines and other services) which parts of your site they may or may not crawl. If crawling is new to you, read how search engines work first.

Think of it as a sign on a staff door: "Staff only." Polite visitors respect the sign. It doesn't lock the door, and it doesn't stop anyone from telling others the room exists.

A few facts from Google's robots.txt documentation are worth knowing:

The file must be named robots.txt and live in the top-level folder of the host. A file at /blog/robots.txt is ignored.
It applies only to the exact host, protocol and port. A subdomain such as shop.example.com needs its own file.
It must be plain text in UTF-8. Google ignores anything beyond 500 KiB.
Google says its main purpose is to avoid overloading your site with requests.

Here is an example of a simple file for an invented online shop:

# Example robots.txt for www.example.com
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search/

Sitemap: https://www.example.com/sitemap.xml

In plain English: "All crawlers, please don't crawl the cart, the checkout or internal search results. My sitemap is here."

How do robots.txt rules work?

Each rule is a line with a field name, a colon and a value. Rules are grouped under a User-agent line, which names the crawler they apply to. Google supports four fields:

Line	What it means	Example
`User-agent`	Which crawler the group is for; `*` means all crawlers	`User-agent: Googlebot`
`Disallow`	A path the crawler should not crawl	`Disallow: /checkout/`
`Allow`	A path it may crawl, usually an exception inside a blocked folder	`Allow: /checkout/help/`
`Sitemap`	The full URL of your sitemap; not tied to any user agent	`Sitemap: https://www.example.com/sitemap.xml`
`#`	A comment for humans; crawlers ignore it	`# Block internal search`

Some other details trip people up:

Paths are case-sensitive. Disallow: /Photos/ does not block /photos/.
Two wildcards are supported. * matches any characters, and $ marks the end of a URL. So Disallow: /*.pdf$ blocks URLs ending in .pdf.
The most specific rule wins. Google uses the rule with the longest matching path. If an Allow and a Disallow rule are equally specific, Google picks the less restrictive one (Allow).
A crawler follows only one group: the one that matches its name most specifically. If you have a Googlebot group, Googlebot ignores the * group completely. That catches many people out.
Unsupported lines are ignored. Google says fields like crawl-delay are not supported, so they do nothing for Googlebot.

The one line that can hurt you most

This rule blocks your entire site for every crawler:

User-agent: *
Disallow: /

It is sometimes added while a site is in development and then forgotten at launch. Compare it with Disallow: with nothing after it, which blocks nothing at all. One slash makes all the difference.

What can't robots.txt do?

This is the most important part of the lesson. Google says it directly: robots.txt is not a way to keep a web page out of Google.

Here's why. Blocking a URL stops Google from crawling it (reading its content). It doesn't stop Google from indexing the URL if other pages link to it. In that case, the URL can still appear in results, usually without a description, because Google couldn't read the page.

There's a second trap. If you add a noindex tag to a page and also block it in robots.txt, Google can't crawl the page, so it never sees the noindex. The two rules cancel each other out. I explain noindex in getting pages indexed (and keeping some out), and our article on robots.txt vs meta robots compares the two in detail.

Also remember that robots.txt is a request, not a lock. Google and other reputable crawlers follow it, but Google notes that not every crawler does. And because the file is public, anyone can read it, so never list secret folders there.

What you want	Right tool	Why not robots.txt?
Stop Google wasting time on endless filter or search URLs	robots.txt	This is exactly what it's for
Keep a page out of search results	`noindex` tag or header	Blocked URLs can still be indexed from links
Hide private or customer data	Password protection / login	robots.txt is public and only a request
Remove a page completely	Delete it (404 or 410), maybe redirect	Blocking doesn't remove anything
Choose between duplicate URLs	Canonical tag or redirect	Google can't see duplicates' signals if it can't crawl them

What should a typical business site block?

Less than you might think. Many small sites need few or no Disallow rules. Google's guidance suggests robots.txt is best used for managing crawl traffic and keeping crawlers away from unimportant or duplicate pages. Common candidates are:

Internal search result pages, which can create endless URLs.
Cart, checkout and account pages in online shops.
Filter and sort URLs that create near-duplicate versions of category pages (see our guide to faceted navigation).
Calendars that generate endless date pages.

And here is what you should not block:

CSS, JavaScript and images that pages need to display properly. Google says you should not block resources if missing them would make pages harder for Google to understand. This matters for JavaScript-heavy sites.
Pages you want to rank. Obvious, but it happens.
Pages you want removed from Google. Use noindex instead, as above.

If your site is very large, crawl efficiency becomes a bigger topic. Our guide to crawl budget explains when it really matters (for most small sites, it doesn't).

Should I block AI crawlers in robots.txt?

This is a newer question, and it's a business decision more than a technical one. Many AI companies publish the names of their crawlers so you can control them in robots.txt. Two examples I have checked in the official documentation:

Google-Extended is a robots.txt token (not a separate bot) that controls whether content Google crawls from your site may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google says it does not affect your inclusion in Google Search and is not a ranking signal.
OpenAI uses separate crawlers. GPTBot collects content that may be used to train its models. OAI-SearchBot is used to show sites in ChatGPT search. OpenAI says each setting is independent, so you can block training but stay visible in ChatGPT search. It also says changes can take about 24 hours to take effect.

Here is an example of a file that allows normal crawling but opts out of AI training:

# Let everyone crawl the site by default
User-agent: *
Allow: /

# Keep content out of Gemini training and grounding (does not affect Google Search)
User-agent: Google-Extended
Disallow: /

# Keep content out of OpenAI model training, but stay in ChatGPT search
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Before you block anything, think about what you want. If you want your brand to be mentioned and cited by AI assistants, blocking their search crawlers works against you. I cover this in the lesson on SEO for ChatGPT, Perplexity and other AI assistants, and our guide on whether to block or allow GPTBot and others lists more bots. Also note that bad bots can fake these names; see how to verify AI crawler requests.

Try it: find, read and test your own robots.txt

This takes about fifteen minutes:

Open your file. Type your domain followed by /robots.txt in a private browser window, for example https://www.yoursite.com/robots.txt. If you see the text, it is public and working.
Look for Disallow: / under User-agent: * or User-agent: Googlebot. If it is there and your site is live, fix it today.
Read each Disallow line and ask: is this an unimportant or duplicate area, or a page I want in Google?
Check you aren't blocking CSS, JavaScript or image folders your pages need.
Add a Sitemap line if one is missing, using the full URL of your sitemap.
Open the robots.txt report in Google Search Console. It shows which robots.txt files Google found, when it last fetched them, and any errors or warnings.
Test important URLs with the URL Inspection tool, which tells you if a page is blocked by robots.txt.
After any change, request a recrawl of the file in the robots.txt report, or allow about a day; Google generally caches it for up to 24 hours.

One more technical detail for your developer. Google treats the server's response to /robots.txt differently by status code. A 4xx error (like 404) is treated as "no robots.txt", so crawling is unrestricted. A 5xx server error makes Google stop crawling the site for the first 12 hours while it retries, then use its last cached copy for up to 30 days. So a broken robots.txt URL can pause crawling, which is worth knowing.

If you use WordPress, Shopify or a similar platform, your platform usually creates the file for you, and SEO plugins often let you edit it from the dashboard.

Questions beginners ask about robots.txt

Does every website need a robots.txt file?

No. If Google can't find a robots.txt file (the server returns a 404), it simply crawls without restrictions. Many small sites are fine that way. A short file is still handy, because it is an easy place to list your sitemap.

Can I use robots.txt to remove a page from Google?

No. Google says robots.txt is not a way to keep a page out of Google. A blocked URL can still be indexed if other pages link to it. Use a noindex tag, password protection or remove the page instead, and make sure the page is not blocked so Google can see the noindex.

Does Google follow crawl-delay?

No. Google supports only user-agent, allow, disallow and sitemap, and says other fields such as crawl-delay are not supported. Some other search engines may read it, but Googlebot ignores it.

If I block Google-Extended, will I disappear from Google Search?

No. Google says Google-Extended does not affect whether a site is included in Google Search and is not used as a ranking signal. It controls whether Google can use your content for training future Gemini models and for grounding in Gemini Apps and Vertex AI.

How quickly does Google notice changes to my robots.txt?

Google generally caches robots.txt for up to 24 hours, so changes can take about a day to apply. You can ask for a faster refresh through the robots.txt report in Search Console.

Using robots.txt safely

Robots.txt is a short public file that asks crawlers to stay out of certain areas. Use it to steer crawlers away from unimportant or endless URLs, list your sitemap, and decide which AI crawlers you welcome. Don't use it to hide or remove pages, never block resources your pages need, and always double-check for a stray Disallow: /.

In the next lesson, XML sitemaps: helping Google find your pages, you will learn how to give Google a clean list of the pages you do want crawled. If you'd like us to review your robots.txt and crawl setup, you can get in touch for a quote.

Robots.txt: Controlling What Search Engines Crawl