Robots.txt for WordPress
Thursday, April 26th, 2007
A simple search for "wordpress robots file" made me realize the importance of robots.txt file. Lots of people seems to be looking for a perfect robots.txt file for WordPress blogs.
After looking at JohnTP's article, I created one that looks like this:
User-agent: *
Disallow: /wp-content/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-
Disallow: /trackback/
Disallow: /cgi-bin/
For them who have never heard of robots.txt, it's basically a file that a web crawler would read to know what they should not crawl at your website.
