Robots.txt is a file that is placed at the root directory of any domain. It is the first file that is read by crawlers on the website. IT is basically a set of protocols that a user can set for different crawlers.
What robots.txt does?
Robots.txt is a derivative that instructs crawlers about which page to crawl and which not to.
What does a robots.txt file contain?
Robots.txt file contains the name of the crawler which is specified by “user-agent” and are divided by a colon. In the next line, the parameters for the crawler are defined. Here is the exact syntax in case of Googlebot, if we want to disallow a particular page.
User-agent: Googlebot
Disallow: *
What are the parameters of crawlers in robots.txt file?
The first parameter is the “user-agent” where one can define the name of the crawlers to specify for which particular crawler the rule is mentioned and meant for.
Second parameter is allow/disallow which instructs crawlers whether they are allowed to read a specific file/folder or not.
Third parameter is “sitemap”. Defining a sitemap in robots.txt is optional
Limitations of Using Robots.txt file
As I just said, Robots.txt is just a derivative and not a command. Crawlers such as Googlebot and Bingbot which are decent ones will definitely follow the protocol defined in the file. But many of the crawlers out there on the internet may not follow it.
Most of the crawlers that do not follow the robots.txt protocol are the deceptive ones, usually the ones hackers use for collecting data. Let’s have a look at the list of limitations of robots.txt file:
It is a derivative that may or may not be followed by crawlers.
It does not block website from appearing at 3rd party websites.
The syntax in robots.txt may be interpreted differently by different types of crawlers.
Meta Robots Tag
Meta robots tag is meta tag that works specifically for the page on which it is defined. It has several options available such as Index, Noindex, Follow, Nofollow, Noarchive, Nosnippet, NOODP, Noimageindex and Notranslate.
Difference between Robots.txt and Meta Robots Tag
Meta Robots tag works quite similar to robots.txt file but has few differences.
As mentioned, robots.txt works site-wide whereas Meta Robots works only on a specific page but has more functions that are limited to that page only. It depends on what is the purpose of using meta robots or robot.txt. You must be clear with the intentions so that you can make a decision when and where one of these can be used.
