Googlebot blocked by robots.txt

Hello,

We apologize for the issue you are facing with Googlebot being blocked by your robots.txt file. Based on the information you’ve provided, it seems that your robots.txt file contains a directive that disallows all user agents from crawling your site:

User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: AhrefsBot
Disallow: /

This directive effectively blocks all specified bots from accessing any URLs on your site. Even if you haven’t added this command in your Rank Math settings, it might be possible that it was added manually or by another plugin.

To resolve this issue, you can follow these steps:

1. Check Your Current robots.txt File: Use the Robots Testing Tool to see the current version of your robots.txt file. Make sure to select the appropriate user agent (e.g., Googlebot) and test your URL.
   

2. Edit Your robots.txt File: If you find the disallow rule, you will need to edit your robots.txt file to remove or modify it. You can refer to this guide on editing robots.txt with Rank Math.

3. Clear Cache: If you have caching enabled on your site, make sure to clear it after making changes to ensure that the updated robots.txt file is served.

4. Re-test in Google Search Console: After making the necessary changes, go back to Google Search Console and re-test your URLs to see if the issue persists.

If you continue to experience difficulties or if the robots.txt file does not reflect the changes you made, please reply to this ticket with your site URL so that a human agent can assist you further.

Thank you for raising this concern, and we hope to help you resolve it soon!

Sources:
Common robots.txt Issues & How to Fix Them » Rank Math
How to Fix ‘Blocked by robots.txt’ Error in Google Search Console » Rank Math