Fix Google AdSense Crawl Errors For WordPress Using Robots.txt File

If you are here on this page to fix Google AdSense crawl errors then that means you’ve got your AdSense account approved. So first of all “Congrats!”

The reason I wished “Congrats” is because getting Google AdSense Approval is not an easy task these days no matter how much efforts you put into starting a blog. Google has tightened their policies because of an increasing number of policies violations.

In case you are still struggling with your AdSense account approval then check this guide: 10+ Powerful Tricks To Get AdSense Approval Fast

Fix Google AdSense Crawl Errors For WordPress

Ok, finally you have an AdSense account. On the first day of approval, you felt pleased. But the problem is that as soon as AdSense starts crawling your site, it shows some crawl errors which affect your “Revenue Optimization” value.

To boost your AdSense income, feel free to check this guide: 11 Revenue Optimization Tricks To Boost Google AdSense Income

To fix this issue, I’m using Robots.txt file method. Just because it is the easiest way to prevent web crawlers and other web robots from accessing all or some part of a website which are otherwise publicly viewable. So we’ll restrict some significant portions of our site like wp-admin, index.php, preview pages etc.

Also Check: 30+ Highest Paying AdSense Niches To Maximize CPC & CTR

For this, add the following given lines to your WordPress site’s Robots.txt file:

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /archives/
Disallow: /comments/feed/
Disallow: /trackback/
Disallow: /index.php
Disallow: /xmlrpc.php
Disallow: *?wptheme
Disallow: ?comments=*
Disallow: /search?
Disallow: /?p=*
Disallow: /feeds
User-agent: Mediapartners-Google*
Allow: /

That’s all!

This method will surely solve your problem as it did for me. Now, wait for at least 24 hours until Google recrawls your site. If you have any doubts or queries regarding this tutorial, then feel free to ask in the comments section below.

Also Check:

Ritesh Saini
Ritesh Saini
I’m a Professional Blogger & Internet Marketer living in India. Started my blogging and online marketing journey in 2013 when I was just 18 years old. You can see my work here on this blog and at

Best Managed WordPress HostingWPX Hosting
Best Managed Cloud HostingCloudways
Best Shared HostingSiteGround
Best Site & Page BuilderElementor
Beautiful WordPress ThemesElegant Themes
Lightest & Fastest WordPress ThemeGeneratePress
Best Lead Generation ToolOptinMonster
Best CDNStackPath
Best Domain RegistrarNamecheap

Related Posts

1 thought on “Fix Google AdSense Crawl Errors For WordPress Using Robots.txt File”

  1. Hi Ritesh, Thanks for this wonderful article. I found it while searching to find solution for the problem m having with my site. It’s about adsense–it’s been almost 3 days since adsense stopped showing up and I couldn’t find any reason, there are no notifications or warning from my account however upon searching, I found through my adsense status section that crawler is unable to access my pages. I wonder if this might be the cause and if so my also adsense stop to all my other sites. I would love to here from you and once again thanks for writing this wonderful piece of information.
    Best Regards.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

ShortPixel Logo 50% Extra Credits
ShortPixel Coupon
Use our exclusive ShortPixel deal and get 50% extra credits on your purchase of one-time credits.
PeerBoard Logo $59 for Lifetime
Get PeerBoard for Lifetime @ $59 ONLY!
PeerBoard lets you create Reddit style forums with ease. It is self-hosted and requires no technical expertise. Grab this lifetime deal before it's gone forever.

Share via
Fix Google AdSense Crawl Errors For WordPress Using Robots.txt File
Scroll to Top
Send this to a friend