What is Robots.txt

It is a text file that a search robot do for your site Search engines will index pages.

In other words, the search engine that will go in search of The robot checks that sometimes, you Some of the changes in the site or whether the update Checking the deeds of the text file robots Help him with the information.

But information that search engine not crawled give by you, Must be set before robots.txt File create.