A passionate programmer’s findings in the world of internet.

Robots.txt for WordPress

Thursday, April 26th, 2007

A simple search for "wordpress robots file" made me realize the importance of robots.txt file. Lots of people seems to be looking for a perfect robots.txt file for WordPress blogs.

After looking at JohnTP's article, I created one that looks like this:

User-agent: *
Disallow: /wp-content/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-
Disallow: /trackback/
Disallow: /cgi-bin/

For them who have never heard of robots.txt, it's basically a file that a web crawler would read to know what they should not crawl at your website.

Check PageRank for ALL Links in Your Page

Thursday, April 20th, 2006

This great search engine optimization tool, PageRank Overlay by SEOBench shows the PageRank of all pages you linked to. Try it!

View the Google PR of ALL the links in the entered URL.

Enter URL

(via Philipp Lenssen)