Google AdSense Robots : All about Mediapartners Google and AdsBot

Google AdSense Robots such as Mediapartners Google and AdsBot works are used by Google to automatically discover and scan content to show relevant AdSense Ads.

How Google AdSense Robots work?

Google AdSense Robots work in the same manner as Google’s main crawler Googlebot works. If they are not blocked, Google AdSense Robots will find out the pages where there you might not want to allow to crawl.

Previously, we wrote about how to configure robots.txt file for GoogleBot and Robots.txt file. It is important to understand how robots.txt file works before understanding how Google AdSense Robots can be controlled too.

According to Google, these following Google AdSense Robots are in use :

Among the Google AdSense Robots, Mediapartners-Google crawls for Google Mobile AdSense and Google AdSense. Whereas, Google AdsBot; AdsBot-Google crawls to check the landing page quality. By default, both these  Google AdSense Robots will crawl all they find if not blocked.

Syntax those Google AdSense Robots follows

Google AdSense Robots follow the same syntax like GoogleBot for search :

User-agent: Googlebot
Disallow: /

User-agent: Mediapartners-Google
Disallow:

You can check from Google Webmaster Tools, in the way we have discussed ago; whether a page is accessible by Google AdSense Robots :

Why to block  Google AdSense Robots from robots.txt?

You might want to save bandwidth that will be used by Google AdSense Robots or to more fine tune robots.txt to show Google AdSense Robots only the web pages where your are using your AdSense Ads.

Is there any need to allow Google AdSense Robots to crawl where there are no Ads? Google clearly says :

This change will allow our crawler to index the content of your site and provide you with the most relevant Google ads for that content.