• Home
  • Archive
  • Tools
  • Contact Us

The Customize Windows

Technology Journal

  • Cloud Computing
  • Computer
  • Digital Photography
  • Windows 7
  • Archive
  • Cloud Computing
  • Virtualization
  • Computer and Internet
  • Digital Photography
  • Android
  • Sysadmin
  • Electronics
  • Big Data
  • Virtualization
  • Downloads
  • Web Development
  • Apple
  • Android
Advertisement
You are here: Home » What are Robots.txt and sitemap

By Abhishek Ghosh March 11, 2011 6:45 am Updated on March 11, 2011

What are Robots.txt and sitemap

Advertisement

Robots.txt

 

For web pages to be indexed, we need programs, robots, scour the net in search of unknown or changed pages to beWhat are Robots.txt and sitemap added to the engine. When it crawls your site, the first thing he wants is a text file robots.txt.

This file can give polite requests to search engine bots. You can tell them they have the right to access your site and if they can index all or a specific page. For this, there are two commands: User-agent and Disallow.

Advertisement

---

 

 

 

 

User-agent can specify the robots which pages are allowed. It can take several forms:

User-agent: * – All robots can index.
User-agent: robot – Only the specified robot can index.

Disallow used to declare the pages that you do not want the engine indexes. It can be used like this:
Disallow: / dir / – A directory will not be indexed.
Disallow: / page.html – Only page.html will not be indexed.

 

You can use several commands to Disallow later. They must each be placed on one line. The robots.txt file must then be inserted at the root of your site.

 

Google provides a tool for automatically generating a robots.txt file on its site in the Webmaster Tools.

 

Define a Sitemap

 

You can also use the Sitemap to indicate the absolute address of a file XML sitemap on your site. This can give:

Sitemap:  http://www . yoursite . com/sitemap.xml

The sitemap is a simple text file that corresponds to the XML standard, which contains all links to your site to allow Google to access it more easily.

 

In Webmaster Tools, you can also use the feature that checks your robots.txt is valid and that the sitemap is detected. You get the number of URLs added to the engine.

 

It is better to add an human readable Sitemap of the site too, preferencially on the footer. The reason is simple, you as a human can not read this xml sitemap of this website easily; but you can easily read and search anything from this version of sitemap of our website.

 

To summarize

 

  • The robots.txt file is the first thing read by a robot when trying to browse your Web pages.
  • User-agent can specify the robot and allowed to choose Disallow directories that should not be indexed.
  • Sitemap command allows to specify where that file must contain all the links in your site.
Signature Tagged With orldhttps://thecustomizewindows com/sitemap/

This Article Has Been Shared 715 Times!

Facebook Twitter Pinterest

Abhishek Ghosh

About Abhishek Ghosh

Abhishek Ghosh is a Businessman, Surgeon, Author and Blogger. You can keep touch with him on Twitter - @AbhishekCTRL.

Here’s what we’ve got for you which might like :

Articles Related to What are Robots.txt and sitemap

  • How to configure Feedburner so that it does not affect your SEO?

    Feedburner is very useful for a publisher of blogs since the free service lets you know how many people subscribe to RSS. Many content management tools used to disseminate an RSS feed, however none of them can get statistics on subscribers.

  • 20 tips to lose your time in SEO

    A humorous writing about the efforts some new web masters spend for doing SEO for their website.

  • 10 techniques to increase the search engine crawling

    We are giving some ideas for increasing the number of engines crawl on your site so that you can get the advantage of relevant content and to position your page on top of search results.

  • Auto blog content generation tools means inviting Google’s penalty

    Auto blog content generation tools are increasingly being used for creating revenue generating blogs. Google however penalizes after discovering auto blogs today or tomorrow.

  • Basic SEO Tips for Bloggers : the SEO kick starter guide

    Basic SEO Tips for Bloggers is targeted for the newbie Webmasters who fails to understand the complex terms at the starting of their blog or website.

Additionally, performing a search on this website can help you. Also, we have YouTube Videos.

Take The Conversation Further ...

We'd love to know your thoughts on this article.
Meet the Author over on Twitter to join the conversation right now!

If you want to Advertise on our Article or want a Sponsored Article, you are invited to Contact us.

Contact Us

Subscribe To Our Free Newsletter

Get new posts by email:

Please Confirm the Subscription When Approval Email Will Arrive in Your Email Inbox as Second Step.

Search this website…

 

Popular Articles

Our Homepage is best place to find popular articles!

Here Are Some Good to Read Articles :

  • Cloud Computing Service Models
  • What is Cloud Computing?
  • Cloud Computing and Social Networks in Mobile Space
  • ARM Processor Architecture
  • What Camera Mode to Choose
  • Indispensable MySQL queries for custom fields in WordPress
  • Windows 7 Speech Recognition Scripting Related Tutorials

Social Networks

  • Pinterest (24.3K Followers)
  • Twitter (5.8k Followers)
  • Facebook (5.7k Followers)
  • LinkedIn (3.7k Followers)
  • YouTube (1.3k Followers)
  • GitHub (Repository)
  • GitHub (Gists)
Looking to publish sponsored article on our website?

Contact us

Recent Posts

  • Cyberpunk Aesthetics: What’s in it Special January 27, 2023
  • How to Do Electrical Layout Plan for Adding Smart Switches January 26, 2023
  • What is a Data Mesh? January 25, 2023
  • What is Vehicular Ad-Hoc Network? January 24, 2023
  • Difference Between Panel Light, COB Light, Track Light January 21, 2023

About This Article

Cite this article as: Abhishek Ghosh, "What are Robots.txt and sitemap," in The Customize Windows, March 11, 2011, January 29, 2023, https://thecustomizewindows.com/2011/03/what-are-robots-txt-and-sitemap/.

Source:The Customize Windows, JiMA.in

PC users can consult Corrine Chorney for Security.

Want to know more about us? Read Notability and Mentions & Our Setup.

Copyright © 2023 - The Customize Windows | dESIGNed by The Customize Windows

Copyright  · Privacy Policy  · Advertising Policy  · Terms of Service  · Refund Policy

We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies.
Do not sell my personal information.
Cookie SettingsAccept
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
SAVE & ACCEPT