Disallow crawler access to wp-admin folder to decrease server load

Here is an article on why you will Disallow crawler access to wp-admin folder of your WordPress installation, how to Disallow crawler access to wp-admin folder and how to test whether you have done the things rightly.

 

How to test whether Disallow crawler access to wp-admin folder is working?

 

First, you can test disallow crawler working or not from Google Webmaster Tools as we have discussed ago. Secondly, you can test very nicely from Google Search itself in which folders your Disallow crawler access is actually working.

Go to Google Search and use this syntax to search (change our URL with your’s) to se if Disallow Crawler is working:

 

site: http://thecustomizewindows.com/wp-admin

 

Obviously, you can check other folders too (like /wp-content) in this way. Here is our result:

 

Why to Disallow crawler access to wp-admin or wp-includes folder?

 

First, we want to allow Google Crawler to crawl and index only our posts, pages etc nicely. If we allow crawler access to these folders, Google Crawler will crawl, waste its time by crawling and indexing these contents. Secondly, probably no one wants to show the world what we have inside these folders in Google search result, therein we need to disallow crawler. Thirdly, by allowing Google crawler to crawl these folders, we can decrease our server load by disallow crawler to access these folders.

 

How to Disallow crawler access to wp-admin or wp-includes folder?

 

You can Disallow crawler access to these folders by adding the following lines to your robots.txt file. Those who are not our regular readers, we will suggest to read this post before to understand the usage to disallow crawler access. Anyway, add these lines and save:

 

Disallow: /wp-admin/
Disallow: /wp-includes/

 

Bonus tips to Disallow crawler

 

Do not surf these folders from your browser with Google Toolbar installed and enabled. Here is the result of doing such: