Hello,
I’m sorry to hear that you’re experiencing issues with crawlers. To address this, you can use the Robots Testing Tool from Google to check your robots.txt file and ensure that there are no directives blocking Googlebot from accessing your site. You can access the tool by following this link: Robots Testing Tool.
Once you’re in the tool, you can enter the URL of your site and select the user-agent you want to simulate, such as Googlebot. This will allow you to test your robots.txt file and make sure that there are no directives blocking Googlebot from accessing your site.
If you find that there are directives in your robots.txt file that are blocking Googlebot from accessing your site, you will need to remove them or replace them with a more permissive one. If you need help editing your robots.txt file, you can refer to our guide on how to edit robots.txt with Rank Math.
If you encounter any issues while trying to resolve this problem, please feel free to reach out to our dedicated support team for further assistance. We’re available 24/7, 365 days a year and we’re here to help.
I hope this helps!
Sources:
Common robots.txt Issues & How to Fix Them » Rank Math
How to Fix ‘Blocked by robots.txt’ Error in Google Search Console » Rank Math
No Information is Available for This Page » Rank Math
Hello,
Thank you for your patience.
You should only use your website URL in Ahrefs tester. However, you are already using our recommended robots.txt rules and it should work fine:
https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/#default-rules
You can use this tool to test your robots.txt as well to determine if there are any issues:
https://technicalseo.com/tools/robots-txt/
Hope that helps.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.
