Comments on: Robots.txt for WordPress http://blog.enrii.com/2007/04/26/robotstxt-for-wordpress/ A passionate programmer's findings in the world of internet. Tue, 07 Sep 2010 04:04:48 +0000 hourly 1 http://wordpress.org/?v=3.0 By: EngLee http://blog.enrii.com/2007/04/26/robotstxt-for-wordpress/comment-page-1/#comment-34242 EngLee Tue, 08 May 2007 09:20:12 +0000 http://blog.enrii.com/2007/04/26/robotstxt-for-wordpress/#comment-34242 Wow, that's an advanced version. Will examine your robots.txt file. Thanks. Wow, that’s an advanced version. Will examine your robots.txt file. Thanks.

]]> By: Wasabi http://blog.enrii.com/2007/04/26/robotstxt-for-wordpress/comment-page-1/#comment-34241 Wasabi Tue, 08 May 2007 08:07:18 +0000 http://blog.enrii.com/2007/04/26/robotstxt-for-wordpress/#comment-34241 You try this: <code> # This rule means it applies to all user-agents User-agent: * # Disallow all directories and files within Disallow: /cgi-bin/ Disallow: /stats/ Disallow: /dh_ Disallow: /about/legal-notice/ Disallow: /about/copyright-policy/ Disallow: /about/terms-and-conditions/ Disallow: /contact/ Disallow: /tag/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/ Disallow: /wp- Disallow: /trackback/ # The Googlebot is the main search bot for google User-agent: Googlebot # Disallow all files ending with these extensions Disallow: /*.php$ Disallow: /*.js$ Disallow: /*.inc$ Disallow: /*.css$ Disallow: /*.gz$ Disallow: /*.wmv$ Disallow: /*.tar$ Disallow: /*.tgz$ Disallow: /*.cgi$ Disallow: /*.xhtml$ # Disallow Google from parsing indididual post feeds and trackbacks.. Disallow: */feed/ Disallow: */trackback/ # Disallow all files with ? in url Disallow: /*?* Disallow: /*? # Disallow all archived monthlies Disallow: /2006/0* Disallow: /2007/0* Disallow: /2006/1* Disallow: /2007/1* # The Googlebot-Image is the image bot for google User-agent: Googlebot-Image # Allow Everything Allow: /* # This is the ad bot for google User-agent: Mediapartners-Google* # Allow Everything Allow: /* # SiTeMap per i motori di ricerca Sitemap: http://siteweb/sitemap.xml </code> You try this:

# This rule means it applies to all user-agents
User-agent: *

# Disallow all directories and files within
Disallow: /cgi-bin/
Disallow: /stats/
Disallow: /dh_
Disallow: /about/legal-notice/
Disallow: /about/copyright-policy/
Disallow: /about/terms-and-conditions/
Disallow: /contact/
Disallow: /tag/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /wp-
Disallow: /trackback/

# The Googlebot is the main search bot for google
User-agent: Googlebot

# Disallow all files ending with these extensions
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.wmv$
Disallow: /*.tar$
Disallow: /*.tgz$
Disallow: /*.cgi$
Disallow: /*.xhtml$

# Disallow Google from parsing indididual post feeds and trackbacks..
Disallow: */feed/
Disallow: */trackback/

# Disallow all files with ? in url
Disallow: /*?*
Disallow: /*?

# Disallow all archived monthlies
Disallow: /2006/0*
Disallow: /2007/0*
Disallow: /2006/1*
Disallow: /2007/1*

# The Googlebot-Image is the image bot for google
User-agent: Googlebot-Image

# Allow Everything
Allow: /*

# This is the ad bot for google
User-agent: Mediapartners-Google*

# Allow Everything
Allow: /*

# SiTeMap per i motori di ricerca
Sitemap: http://siteweb/sitemap.xml

]]> By: Edwin http://blog.enrii.com/2007/04/26/robotstxt-for-wordpress/comment-page-1/#comment-34229 Edwin Tue, 08 May 2007 02:22:49 +0000 http://blog.enrii.com/2007/04/26/robotstxt-for-wordpress/#comment-34229 Thanks, I added the robots.txt to my blog. Thanks, I added the robots.txt to my blog.

]]>