Free SEO audit for new clients — claim yours
Home/Blog/SEO/Robots.txt vs Meta Robots
SEO

Robots.txt vs Meta Robots

Robots.txt vs Meta Robots
Table of contents
  1. What is Robots.txt?
    1. What robots.txt does?
    2. What does a robots.txt file contain?
    3. What are the parameters of crawlers in robots.txt file?
  2. Limitations of Using Robots.txt file
  3. Meta Robots Tag
  4. Difference between Robots.txt and Meta Robots Tag
  5. Which one out of these should be used?

What is Robots.txt?

Robots.txt is a file that is placed at the root directory of any domain. It is the first file that is read by crawlers on the website. IT is basically a set of protocols that a user can set for different crawlers.

What robots.txt does?

Robots.txt is a derivative that instructs crawlers about which page to crawl and which not to.

What does a robots.txt file contain?

Robots.txt file contains the name of the crawler which is specified by “user-agent” and are divided by a colon. In the next line, the parameters for the crawler are defined. Here is the exact syntax in case of Googlebot, if we want to disallow a particular page.

User-agent: Googlebot

Disallow: *

What are the parameters of crawlers in robots.txt file?

  • The first parameter is the “user-agent” where one can define the name of the crawlers to specify for which particular crawler the rule is mentioned and meant for.
  • Second parameter is allow/disallow which instructs crawlers whether they are allowed to read a specific file/folder or not.
  • Third parameter is “sitemap”. Defining a sitemap in robots.txt is optional

Limitations of Using Robots.txt file

As I just said, Robots.txt is just a derivative and not a command. Crawlers such as Googlebot and Bingbot which are decent ones will definitely follow the protocol defined in the file. But many of the crawlers out there on the internet may not follow it.

Most of the crawlers that do not follow the robots.txt protocol are the deceptive ones, usually the ones hackers use for collecting data. Let’s have a look at the list of limitations of robots.txt file:

  • It is a derivative that may or may not be followed by crawlers.
  • It does not block website from appearing at 3rd party websites.
  • The syntax in robots.txt may be interpreted differently by different types of crawlers.

Meta Robots Tag

Meta robots tag is meta tag that works specifically for the page on which it is defined. It has several options available such as Index, Noindex, Follow, Nofollow, Noarchive, Nosnippet, NOODP, Noimageindex and Notranslate.

Difference between Robots.txt and Meta Robots Tag

Meta Robots tag works quite similar to robots.txt file but has few differences.

Robots.txt Meta Robots
Is applicable site-wide Is applicable on a webpage
Is a derivative Is a tag
Limited options available A number of options are available
Options: Allow/Disallow only Options: Index, Noindex, Follow, Nofollow, Noarchive, Nosnippet, NOODP, Noimageindex, Notranslate
Is case sensitive Not case sensitive
Robots.txt vs Meta Robots comparison

Which one out of these should be used?

As mentioned, robots.txt works site-wide whereas Meta Robots works only on a specific page but has more functions that are limited to that page only. It depends on what is the purpose of using meta robots or robot.txt. You must be clear with the intentions so that you can make a decision when and where one of these can be used.

Rohit Choudhary
Written byRohit ChoudharyCo-Founder, My SEO Experts

Rohit Choudhary is the co-founder of My SEO Experts and has more than 14 years of hands-on experience in SEO, paid advertising and digital marketing. Since starting out in 2011 he has helped hundreds of businesses grow through practical, data-driven search strategies, and he also serves as a consulting Digital Marketing Director for several US-based companies.

Keep reading

Related articles

Ready to grow your business online?

Get a free, no-obligation proposal tailored to your goals and budget.

CallFree Quote