Hello,
We apologize for the issue you are facing with your robots.txt file and the indexing problems it has caused. To help you resolve this, we recommend following these steps:
-
Access your robots.txt file: You can find it at https://yourdomain.com/robots.txt`.
-
Review the file: Check for any Disallow directives that might be blocking Googlebot from crawling your site. If you find any that are preventing access to important pages, consider removing or modifying them.
-
Test your robots.txt file: Use the Robots Testing Tool to see if your current robots.txt file is blocking Google from crawling your site. Make sure to select the “All (robots.txt)” option as the User Agent and click the TEST button.
-
Clear your website cache: If you’ve made changes to your robots.txt file, ensure you clear your website cache and test again to see the latest version.
-
Check for common issues: If you encounter errors like "Blocked by robots.txt" in Google Search Console, refer to our guide on fixing indexing issues for more detailed steps.
If you continue to experience issues after following these steps, please reply to this ticket, and a human agent from our support team will assist you further.
Thank you for raising this concern! We hope to help you resolve it soon!
Sources:
Common robots.txt Issues & How to Fix Them » Rank Math
How to Fix Indexing Issues in Google Search Console » Rank Math
Hi,
I am a bit stumped, I have tried everything in the article which you sent, I do not have rank math pro and unfortunately, I am not in a position to buy it just yet.
The url which isn’t indexing is: https://globetrottingmoms.com/wheelchairing-it-in-west-palm-beach/
It looks OK when I run tests via the Rank Math test, but when I run it through Google Search Console, it tells me the robots.txt blocks it. Should I maybe delete Rank Math and reinstall it?
Any advice would be much appreciated.
Many thanks – Heather
Hi Rank Math,
I would appreciate it if someone could get back to me. I don’t know what to do. For the first time in like the lifetime of this blog, it was beginning to gain some traction. I don’t want to lose that because of something like this. I simply can’t figure it out myself and have tried all the options suggested.
Much appreciated, many thanks – Heather
Hello,
Apologies for the delay as we prioritize older tickets, and whenever you post a reply, your ticket gets bumped to the top.
However, your robots.txt rules are set to disallow bots to crawl your site.
To fix this, please follow this guide:
https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/
And here are the default and recommended rules you should have in your robots.txt:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://globetrottingmoms.com/sitemap_index.xml
Once done, clear your website’s cache and revalidate the issue in your Google Search Console account.
Looking forward to helping you.
Hi,
That is exactly the code that is in there. But it’s not working? I do not have noindex on the post I am referring to. And now there appears to be more posts/pages that are being blocked by robots.txt.
Please can you look into this further, whilst I am not a techie, I can follow instructions and I have checked everything in links given, so it’s something else.
Much appreciated. Many thanks – Heather
Hello,
Your robots.txt is already configured.
When we checked it before, it only showed this rule: Disallow: /
However, the link you shared is already indexed on the search results page
If you’re still getting that error in your Google Search Console, please clear your website’s cache, revalidate the issue, and give Google some time to recrawl your site.
Looking forward to helping you.
Yes, I created a robots.txt file and added it to my root directory.
Waiting days for an answer wasn’t an option for something so important. Such a shame as I would rank Rank Math as one of my must-have plug-ins and I promote it widely in blogging circles. However, with this level of support, I will struggle to do so in future. I understand that being on the free version doesn’t give me the same level as pro, but I have bought into the content AI – and I would have hoped that if our blog took off, I would have bought the full version. It’s hard for a blog to take off if it’s not even being indexed!!
Hello,
We apologise for the delay. We are glad to know that this issue has been resolved. Thank you for letting us know.
This ticket will be closed now, but you can always open a new one if you have any other questions or concerns. We are here to help you with anything related to Rank Math.
We appreciate your patience and cooperation throughout this process.
Thank you for choosing Rank Math.
