Fix Google AdSense Crawl Errors For WordPress Using Robots.txt File

If you are here on this page to fix Google AdSense crawl errors then that means you’ve got your AdSense account approved. So first of all “Congrats!”

The reason I wished “Congrats” is because getting Google AdSense Approval is not an easy task these days no matter how much efforts you put into starting a blog. Google has tightened their policies because of an increasing number of policies violations.

In case you are still struggling with your AdSense account approval then check this guide: 10+ Powerful Tricks To Get AdSense Approval Fast

Fix Google AdSense Crawl Errors For WordPress

Ok, finally you have an AdSense account. On the first day of approval, you felt pleased. But the problem is that as soon as AdSense starts crawling your site, it shows some crawl errors which affect your “Revenue Optimization” value.

To boost your AdSense income, feel free to check this guide: 11 Revenue Optimization Tricks To Boost Google AdSense Income

To fix this issue, I’m using Robots.txt file method. Just because it is the easiest way to prevent web crawlers and other web robots from accessing all or some part of a website which are otherwise publicly viewable. So we’ll restrict some significant portions of our site like wp-admin, index.php, preview pages etc.

Also Check: 30+ Highest Paying AdSense Niches To Maximize CPC & CTR

For this, add the following given lines to your WordPress site’s Robots.txt file:

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /archives/
Disallow: /comments/feed/
Disallow: /trackback/
Disallow: /index.php
Disallow: /xmlrpc.php
Disallow: *?wptheme
Disallow: ?comments=*
Disallow: /search?
Disallow: /?p=*
Disallow: /feeds
User-agent: Mediapartners-Google*
Allow: /

That’s all!

This method will surely solve your problem as it did for me. Now, wait for at least 24 hours until Google recrawls your site. If you have any doubts or queries regarding this tutorial, then feel free to ask in the comments section below.

Also Check:

1 thought on “Fix Google AdSense Crawl Errors For WordPress Using Robots.txt File”

  1. Avatar

    Hi Ritesh, Thanks for this wonderful article. I found it while searching to find solution for the problem m having with my site. It’s about adsense–it’s been almost 3 days since adsense stopped showing up and I couldn’t find any reason, there are no notifications or warning from my account however upon searching, I found through my adsense status section that crawler is unable to access my pages. I wonder if this might be the cause and if so my also adsense stop to all my other sites. I would love to here from you and once again thanks for writing this wonderful piece of information.
    Best Regards.

Leave a Comment

Your email address will not be published. Required fields are marked *

Share via
Fix Google AdSense Crawl Errors For WordPress Using Robots.txt File
Scroll to Top
Send this to a friend