Hello,
Thank you for reaching out to us with this issue. It seems that Google is encountering difficulties crawling your robots.txt file despite it being accessible through the provided link.
To address this problem, you can try the following steps:
-
Check Robots.txt Content: Ensure that the content of your robots.txt file is correctly formatted and does not contain any directives that might be blocking Googlebot from crawling your site.
-
Test Robots.txt: You can use the Robots Testing Tool to validate your robots.txt file. Enter your website URL, select "All (robots.txt)" as the User Agent, and click the TEST button to see if there are any issues with the file.
-
Review Google Search Console Settings: Double-check your Google Search Console settings to ensure that there are no specific configurations preventing Google from crawling your robots.txt file.
If after following these steps the issue persists, please reply to this ticket, and a human agent from our support team will assist you further.
Thank you for your understanding and patience.
Best regards.
Sources:
How to Fix ‘Blocked by robots.txt’ Error in Google Search Console » Rank Math
Common robots.txt Issues & How to Fix Them » Rank Math
Hello,
We’ve checked your robots.txt file, and we can see it is working properly without any issues.
In this case, please try to clear your website cache including any server-level cache, and check the robots.txt again in Google Search Console.
If the issue persists, please create a static robots.txt file in your website’s root directory with the same rules.
Once done, clear your website’s cache and monitor your Google Search Console.
Looking forward to helping you.
I have created a new robots.txt and it has update on the link with the new contents https://www.taxleaders.net/robots.txt
Still get the same errors. Website cache has also been cleared.
Also, every online robots.txt validator is detecting my robots.txt except Google.
Hello,
Google is reporting a sitewide issue. It means something on your site or server is blocking Googlebot from fetching the robots.txt file. You should check with your host and ask them to whitelist Googlebot in your site & server’s firewall.
Let us know how that goes. Looking forward to your update.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.
