Skip to content
stated:methodYour own data, read and written up

AdSense crawler and robots.txt

Most robots.txt files speak to every crawler at once, under User-agent: *. The crawler AdSense uses is one Google says does not read that group at all, so what your file tells it is only what your file tells it by name.

Which crawler it is

Its user agent token is Mediapartners-Google. Google describes it as visiting participating sites in order to provide them with relevant ads, and says that preferences addressed to it affect ad-related products, AdSense and Ad Manager among them. On desktop it identifies itself simply as Mediapartners-Google; on mobile the same name appears, as Mediapartners-Google/2.1, inside a longer string.

Does the AdSense crawler follow User-agent: * rules

No. Google's documentation of its special-case crawlers says of Mediapartners-Google, the crawler AdSense uses, that the global user agent (*) is ignored. It follows only the rules in a group that names it, so a Disallow written for every crawler does not keep it out, and a Disallow in a group addressed to Mediapartners-Google does. AdSense's own help page on site access says that when such a group disallows the whole site, with Disallow: /, Google can't serve ads on it.

Three files, read the way it reads them

User-agent: *
Disallow: /
Every crawler that obeys the global group is kept out of the whole site. The AdSense crawler is not, because the global group is the one it ignores.
User-agent: Mediapartners-Google
Disallow: /
The AdSense crawler is kept out of the whole site. These are the two lines AdSense's help page tells you to remove, saying that while they stand Google can't serve ads on the site.
User-agent: Mediapartners-Google
Disallow: /services/
It is kept out of everything under /services/ and can read the rest. A section holding most of a site's pages can be closed to it this way while the home page stays open, which is the case the example review describes.

Checking your own

Open your site's robots.txt in a browser, at the root of the domain, and look for a user-agent line naming Mediapartners-Google. Google's robots.txt rules treat both the field name and the crawler's name as case-insensitive and make the space after the colon optional, so user-agent:mediapartners-google names it exactly as User-agent: Mediapartners-Google does.

If no line names it in any form, nothing in the file is addressed to the AdSense crawler, whatever the lines under User-agent: * say. If one does, the Allow and Disallow lines in that group are what it obeys, and where more than one group names it, Google combines their rules into one. A group can also name several crawlers on consecutive user-agent lines, with its rules applying to each of them, so the name can sit in a group that was written with another crawler in mind.

AdSense's help page says that changes to the file might not be reflected in its index.

What this does not tell you

Both Google pages this rests on are about crawling and about serving ads. Neither says how robots.txt bears on a site review, so a group that shuts the AdSense crawler out is something to fix rather than an established reason for a rejection. A review also weighs what no file can show. Google says the specific reason is in your AdSense account or the email it sent, and a duplicate account, an identity or payment problem, and traffic sources are not visible from the public site.

The AdSense rejection review reads a site's robots file for exactly this, alongside the rest of what a public read can see, and the example review shows the case in its section 03.

Sources

Google Search Central, Google's special-case crawlers, for the user agent token, what the crawler does and that the global user agent is ignored. AdSense Help, the site access page on robots.txt, for the two lines it tells you to remove and what happens to ads while they stand. Both were read on 17 October 2026.