A robot.txt file simply functions as a request that is ignored by specific robots as well as the directories in their search. The robot.txt files are used to provide privacy for many search engine results for the fact that content or information might prove misleading to the visitor.
About the Author
10 Strong Reasons Your Business Needs SEO August 26, 2016
Battle of the Bots – An Overview of Different Facets of Bots February 15, 2016