Go Back   Technibble Forums > Advertising & Marketing > Website Development & SEO

  Technibble Sponsor

 
 
Thread Tools Display Modes
Prev Previous Post   Next Post Next
  #1  
Old 03-12-2012, 11:27 AM
TLE's Avatar
TLE TLE is offline
 
Join Date: May 2009
Location: UK
Posts: 1,239
TLE is on a distinguished road
Default Wordpress Robots.txt file content

I have noticed quite a bit of duplicated data being indexed on my Wordpress website which is going to dilute any SEO work done. I have also noticed a few directorys being indexed which is bad practice from a security point of view.

Here is what I have so far... have I missed anything or blocked somthing I shouldn't have. I really only want my main pages indexed.

Code:
User-agent: SiteSnagger
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /wp-content/uploads/lg-gallery/
Disallow: /wp-content/uploads/
Disallow: /wp-content/themes/zen/audio_player/
Disallow: /wp-login.php
Disallow: /*wp-login.php*
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /author
Disallow: /contact/
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /category/*
Disallow: /category/
Disallow: /login/


Sitemap: http://www.thirsk-homeitsolutions.co.uk/sitemap.xml.gz
Reply With Quote
 

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 02:41 PM.


Powered by vBulletin®
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Technibble.com is based out of MELBOURNE, AUSTRALIA.