The robot.txt file is what search engine spiders look at before indexing your site. This file tells the search engines what files and directories to ignore.
This is great to stop search engines putting pages in the index that are not needed. Like the tell a friend page or the contact us page.
If you are using [...]