The robot.txt file is what search engine spiders look at before indexing your site. This file tells the search engines what files and directories to ignore.

This is great to stop search engines putting pages in the index that are not needed. Like the tell a friend page or the contact us page.

If you are using the Ultimate SEO URL plug-in and zen 1.3.7 then replace the contents of your robots.exe file with the following

User-agent: *
Disallow: /cache/
Disallow: /contrib/
Disallow: /docs/
Disallow: /extras/
Disallow: /htmlarea/
Disallow: /stats/
Disallow: /sitemap/
Disallow: /graphics/
Disallow: /login.html
Disallow: /privacy.html
Disallow: /conditions.html
Disallow: /contact_us.html
Disallow: /gv_faq.html
Disallow: /discount_coupon.html
Disallow: /unsubscribe.html
Disallow: /shopping_cart.html
Disallow: /ask_a_question.html
Disallow: /popup_image_additional.html
Disallow: /product_reviews_write.html
Disallow: /tell_a_friend.html
Disallow: /pages-popup_image.html?
Disallow: /popup_image_additional.html?
Disallow: /login.html
User-agent: Googlebot-Image
Disallow: /

The last bit “Googlebot-Image” stops goggle from indexing your pictures into the photo search. If you want your photos in the Google photo search remove the last two lines.

Using this robots.txt file will help to prevent “diluting” your site, which may help your search engine positions.

You can easily modify this list if you are not using Ultimate SEO URLs or a different SEF urls plugin.

**Contributed by Scott** Do not add your admin folder to this list. See comments

Share and Enjoy:
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Google
  • BlinkList
  • description
  • StumbleUpon
  • Technorati