What are Robots.txt and sitemap

Robots.txt

For web pages to be indexed, we need programs, robots, scour the net in search of unknown or changed pages to be added to the engine. When it crawls your site, the first thing he wants is a text file robots.txt.

This file can give polite requests to search engine bots. You can tell them they have the right to access your site and if they can index all or a specific page. For this, there are two commands: User-agent and Disallow.

User-agent can specify the robots which pages are allowed. It can take several forms:

User-agent: * – All robots can index.
User-agent: robot – Only the specified robot can index.

Disallow used to declare the pages that you do not want the engine indexes. It can be used like this:
Disallow: / dir / – A directory will not be indexed.
Disallow: / page.html – Only page.html will not be indexed.

You can use several commands to Disallow later. They must each be placed on one line. The robots.txt file must then be inserted at the root of your site.

Google provides a tool for automatically generating a robots.txt file on its site in the Webmaster Tools.

Define a Sitemap

You can also use the Sitemap to indicate the absolute address of a file XML sitemap on your site. This can give:

Sitemap:  http://www . yoursite . com/sitemap.xml

The sitemap is a simple text file that corresponds to the XML standard, which contains all links to your site to allow Google to access it more easily.

In Webmaster Tools, you can also use the feature that checks your robots.txt is valid and that the sitemap is detected. You get the number of URLs added to the engine.

It is better to add an human readable Sitemap of the site too, preferencially on the footer. The reason is simple, you as a human can not read this xml sitemap of this website easily; but you can easily read and search anything from this version of sitemap of our website.

To summarize

• The robots.txt file is the first thing read by a robot when trying to browse your Web pages.
• User-agent can specify the robot and allowed to choose Disallow directories that should not be indexed.
• Sitemap command allows to specify where that file must contain all the links in your site.

0saves

Save

If you enjoyed this post, please consider leaving a comment or subscribing to the RSS feed to have future articles delivered to your feed reader.

Visitors Read This Post Also Read:

1. 15 Tips to secure your WordPress site Some time ago we had a problem with our WordPress site suffered a little hacking....
2. Tips and tricks to start and continue a professional and successful blog In this article we will give you some tips for a having a successful  and...
3. 100 tips for success on the Internet! We have compiled this guide from our experience in the Internet, that is our best...
4. What is the professional way of use of Twitter? Anyone has heard of Twitter, the microblogging service that allows you to send messages of...
5. Microsoft will integrate tracking protection in Internet Explorer Microsoft plans to give users of its new internet browser the ability to stop certain...
6. Edit the hosts file in Windows 7 One of the functions of the Hosts file is to block or allow Internet pages...
7. Create a favicon for your WordPress blog Branding is a very important factor on the web to differentiate yourself from others, to...
8. Various ways to create a website By using software   If you want to create a site quickly, the best way...
9. Create a custom login page in WordPress This tutorial will show you how to set up a custom login page in WordPress....
10. SEO advantages of using url shortener services There are several shortener services or shortening urls shortener of urls as, among them most...
11. Guide to Create a 3rd Party Theme We are not talking about changing the wallpapers, changing the taskbar color , saving it...
12. Dictionary of DLL, VXD, OCX files In this tutorial we are offering a small dictionary of the DLL, VXD, OCX and...
13. The principle and how to choose a browser The principle of the browsers A browser is a software that allows you to view...
14. How to disable file indexing in Windows 7 What is more annoying when you open a folder containing many multimedia files, having to...
15. Disable WordPress administrator bar under 3.1 for all users The 3.1 version of WordPress adds an administrator bar directly on the front end of...