• Home
  • Archive
  • Tools
  • Contact Us

The Customize Windows

Technology Journal

  • Cloud Computing
  • Computer
  • Digital Photography
  • Windows 7
  • Archive
  • Cloud Computing
  • Virtualization
  • Computer and Internet
  • Digital Photography
  • Android
  • Sysadmin
  • Electronics
  • Big Data
  • Virtualization
  • Downloads
  • Web Development
  • Apple
  • Android
Advertisement
The Customize Windows > Wordpress and SEO > SEO tips > What are Robots.txt and sitemap

By Abhishek Ghosh March 11, 2011 6:45 am Updated on March 11, 2011

What are Robots.txt and sitemap

Advertisement

Robots.txt

 

For web pages to be indexed, we need programs, robots, scour the net in search of unknown or changed pages to beWhat are Robots.txt and sitemap added to the engine. When it crawls your site, the first thing he wants is a text file robots.txt.

This file can give polite requests to search engine bots. You can tell them they have the right to access your site and if they can index all or a specific page. For this, there are two commands: User-agent and Disallow.

 

 

 

 

User-agent can specify the robots which pages are allowed. It can take several forms:

User-agent: * – All robots can index.
User-agent: robot – Only the specified robot can index.

Disallow used to declare the pages that you do not want the engine indexes. It can be used like this:
Disallow: / dir / – A directory will not be indexed.
Disallow: / page.html – Only page.html will not be indexed.

 

You can use several commands to Disallow later. They must each be placed on one line. The robots.txt file must then be inserted at the root of your site.

 

Google provides a tool for automatically generating a robots.txt file on its site in the Webmaster Tools.

 

Define a Sitemap

Advertisement

---

 

You can also use the Sitemap to indicate the absolute address of a file XML sitemap on your site. This can give:

Sitemap:  http://www . yoursite . com/sitemap.xml

The sitemap is a simple text file that corresponds to the XML standard, which contains all links to your site to allow Google to access it more easily.

 

In Webmaster Tools, you can also use the feature that checks your robots.txt is valid and that the sitemap is detected. You get the number of URLs added to the engine.

 

It is better to add an human readable Sitemap of the site too, preferencially on the footer. The reason is simple, you as a human can not read this xml sitemap of this website easily; but you can easily read and search anything from this version of sitemap of our website.

 

To summarize

 

  • The robots.txt file is the first thing read by a robot when trying to browse your Web pages.
  • User-agent can specify the robot and allowed to choose Disallow directories that should not be indexed.
  • Sitemap command allows to specify where that file must contain all the links in your site.
Signature
Tagged With orldhttps://thecustomizewindows com/sitemap/

This Article Has Been Shared 6384 Times!

Facebook Twitter Google+ Pinterest
Abhishek Ghosh

About Abhishek Ghosh

Abhishek Ghosh is a Businessman, Orthopaedic Surgeon, Author and Blogger. You can keep touch with him on Twitter - @AbhishekCTRL.

Here’s what we’ve got for you which might like :

Articles Related to What are Robots.txt and sitemap

  • How to configure Feedburner so that it does not affect your SEO?

    Feedburner is very useful for a publisher of blogs since the free service lets you know how many people subscribe to RSS. Many content management tools used to disseminate an RSS feed, however none of them can get statistics on subscribers.

  • 20 tips to lose your time in SEO

    A humorous writing about the efforts some new web masters spend for doing SEO for their website.

  • 10 techniques to increase the search engine crawling

    We are giving some ideas for increasing the number of engines crawl on your site so that you can get the advantage of relevant content and to position your page on top of search results.

  • Auto blog content generation tools means inviting Google’s penalty

    Auto blog content generation tools are increasingly being used for creating revenue generating blogs. Google however penalizes after discovering auto blogs today or tomorrow.

  • Basic SEO Tips for Bloggers : the SEO kick starter guide

    Basic SEO Tips for Bloggers is targeted for the newbie Webmasters who fails to understand the complex terms at the starting of their blog or website.

Additionally, performing a search on this website can help you. Also, we have YouTube Videos.

Take The Conversation Further ...

We'd love to know your thoughts on this article.
Meet the Author over on Twitter to join the conversation right now!

If you want to Advertise on our Article or want a Sponsored Article, you are invited to Contact us.

Contact Us



Subscribe To Our Free Newsletter

You can subscribe to our Free Once a Day, Regular Newsletter by clicking the subscribe button below.

Click To Subscribe

Please Confirm the Subscription When Approval Email Will Arrive in Your Email Inbox as Second Step.

Search this website…

 

Popular Articles

Our Homepage is best place to find popular articles!

Here Are Some Good to Read Articles :

  • Cloud Computing Service Models
  • What is Cloud Computing?
  • Cloud Computing and Social Networks in Mobile Space
  • ARM Processor Architecture
  • What Camera Mode to Choose
  • Indispensable MySQL queries for custom fields in WordPress
  • Windows 7 Speech Recognition Scripting Related Tutorials

Social Networks

  • Pinterest (15K Followers)
  • Twitter (4.4k Followers)
  • Facebook (5.2k Followers)
  • LinkedIn (3.3k Followers)
  • YouTube (1.5k Followers)
  • GitHub (Repository)
  • GitHub (Gists)
Looking to publish sponsored article on our website?

Contact us

Recent Posts

  • 6 Tips for Securing Your Data from Cyber Attacks as a Remote Worker December 11, 2019
  • What is Process Mining? December 10, 2019
  • Explanation of the ESP32 Vulnerability Warnings December 6, 2019
  • OLTP versus OLAP December 4, 2019
  • What is OLTP (Online Transaction Processing)? December 3, 2019

About This Article

Title: What are Robots.txt and sitemap
2011-03-11

Author: Abhishek Ghosh
Subjects: SEO tips

Is Part Of:


TheCustomizeWindows,

Friday, March 11th, 2011,
Vol.1(01),
p.1–3863 [IoT Ready Journal]


Source:The Customize Windows
ISSN: 0019-5847 ;
E-ISSN: 0019-5847 ;
Publisher:
jima.in

Cite this article as: Abhishek Ghosh, "What are Robots.txt and sitemap," in The Customize Windows, March 11, 2011, December 11, 2019, https://thecustomizewindows.com/2011/03/what-are-robots-txt-and-sitemap/.

This website uses cookies. If you do not want to allow us to use cookies and/or non-personalized Ads, kindly clear browser cookies after closing this webpage.

Read Cookie Policy.


PC users can consult Corrine Chorney for Security.

Want to know more about us? Read Notability and Mentions & Our Setup.

web analysis

Copyright © 2019 - The Customize Windows | dESIGNed by The Customize Windows

Copyright  · Privacy Policy  · Advertising Policy  · Terms of Service  · Refund Policy