What is Robot txt File

Robot.Txt File Solution and Explanation

what is Robot.txt file?

Webmasters can direct web robots (also known as spiders or crawlers) how to navigate their site by putting instructions in a plain text file called robots.txt. It specifies to web robots which parts of a website should be crawled and indexed and which should be left alone during the indexing process.

The robots.txt file always has the name "robots.txt" in its filename, and it is always located in the root directory of a website. Entering the file's Uniform Resource Locator (URL) in a web browser will allow anyone to view it as long as they have access to the internet.

Web robots, such as search engine crawlers, website analyzers, and other automated tools, refer to the instructions contained in a website's robots.txt file in order to figure out which pages of the website to crawl and index. Webmasters have the ability to exert control over the manner in which web robots interact with their website and block certain pages, directories, or files from being indexed or crawled by utilizing the robots.txt file.

why we use robot.txt file?

Robot.txt file use to communicate with web robots, spiders or crawlers.

The main purpose of the robots.txt file is to stop online robots, such the ones used by search engine crawlers, from scanning and indexing particular web pages or parts of a website. By blocking web robots from accessing the file, this may be achieved.

By requesting that web robots not crawl and index particular pages or portions of a website, webmasters may conserve bandwidth and server resources. The website may perform better as a result of this.

To protect a website's privacy, it is feasible to utilize the robots.txt file to limit access to particular pages or parts. There may be private or confidential material on some pages or sections.

and also use 

• To control access to certain parts of a website
• To improve website performance
• To prevent duplicate content issues
• To comply with legal requirements

where it is important?

Robot.txt file is an important tool for website owner to control their website is accessed and indexed by web spiders or crawlers.

The importance of Robot.txt file is for 

• SEO reason
• Security reason
• Web Performance
• compliance

Best robot.txt file example

Here is the Best example for Robot.txt file (specially for blogger website)

you can easily just copy and paste into your blogger Robot file, just replace the website name and save it.

This format will index only your Post, Article and Important Pages.

Example 1:

User-agent: Googlebot

Disallow: /nogooglebot/

User-agent: *

Allow: /

Sitemap:https://yoursite.com/sitemap.xml

Example 2:

User-agent: *

Disallow: 

Sitemap: https://www.yoursite.com/sitemap.xml

Example 3:

User-agent: *

Disallow: /search

Allow: /

Sitemap: https://www.yourblogname.com/sitemap.xml

Note :

In this example No2 the google bot will crawl your whole site and index fast even your content is copyright.