‘Blocked by robots.txt’ error

Hello,

We apologize for the issue you are facing with the ‘Blocked by robots.txt’ error on your website. This error indicates that your robots.txt file is preventing Googlebot from crawling your pages, which can indeed affect your blog posts’ visibility in search results.

Here are some steps you can take to troubleshoot and potentially resolve this issue:

1. Check Your robots.txt File: Ensure that your robots.txt file does not contain any directives that block Googlebot from accessing your content. A basic robots.txt file should look something like this:

   User-agent: *
   Disallow:

   This configuration allows all search engines to crawl all pages.

2. Use Google Search Console: You can find the specific pages that are blocked by going to the Pages section in Google Search Console and clicking on the Not indexed section. Here’s how:

   • Navigate to the Pages section.
   • Click on Not indexed.
   • Look for the Blocked by robots.txt error and click on it to see which pages are affected.
     

3. Check Rank Math’s Analytics: If you have Rank Math PRO, you can also check the Index Status report in Rank Math’s Analytics:

   • Go to Rank Math SEO → Analytics in your WordPress dashboard.
   • Click on the Index Status tab to see the status of your pages.
     

4. Edit Your robots.txt File: If you find any disallow directives that are blocking important pages, you will need to edit your robots.txt file accordingly. You can do this through your WordPress dashboard or via FTP.

5. Regular Review: To prevent this error from happening again, regularly review your robots.txt file and ensure that it is configured correctly.

If you have followed these steps and are still experiencing issues, please reply to this ticket, and a human agent from the Rank Math Support team will assist you further. Thank you for raising this concern, and we hope to help you resolve it soon!

Hello,

In this case, you may update your robots.txt rules to ensure that those URLs are not getting blocked.

You may refer to our default and recommended robots.txt rules here:
https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/#default-rules

Looking forward to helping you.

Hello,

The URLs are not being blocked by the robots.txt file: https://search.google.com/test/rich-results/result?id=Gl9qbL89XAKazOxlOeyYbw

It seems GSC is reporting a false error. Please use the URL Inspection tool and use the Live Test option to check the current status of the URLs.

Let us know how that goes. Looking forward to helping you.