Hello,
I’m sorry to hear that you’re experiencing issues with your website’s URLs not being crawled. There could be several reasons why this is happening, but one common cause is the robots.txt file. It’s possible that the robots.txt file is blocking Google from crawling your website.
To check and edit your robots.txt file using Rank Math, you can navigate to WordPress Dashboard → Rank Math SEO → General Settings → Edit robots.txt. From there, you can ensure that there are no directives that are blocking Googlebot from accessing your site.
If you find any directives in your robots.txt file that are blocking Googlebot, you will need to remove them or replace them with a more permissive one.
Additionally, you can use the robots.txt testing tool from Google to check your file and make sure that there are no directives that are blocking Googlebot from accessing your site.
If you need further assistance, please feel free to reach out to our support team directly from here, and we’re always here to help.
I hope this helps! If you have any other questions or concerns, please don’t hesitate to ask.
Sources:
Common robots.txt Issues & How to Fix Them » Rank Math
Discovered – Currently Not Indexed in Google Search Console » Rank Math
How to Fix “Crawled – Currently Not Indexed” Error in Google Search Console » Rank Math
Why Did My Website Disappear From Google Search? » Rank Math
What are the issues please identify and let us know
Hello,
Thank you for your patience.
We’ve checked your site using the site: operator and your URLs are already included on the search results page
However, your robots.txt contains a disallow rule which could be the cause of this issue.
Please follow our guide here to update your robots.txt to the default and recommended rules:
https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/#default-rules
Moreover, you should check your Google Search Console coverage section to determine any issues reported on your URLs that may have prevented your site from being indexed:
https://support.artlogic.net/hc/en-gb/articles/360016851260-Google-Search-Console-errors-and-coverage-issues
Hope that helps.
Hi,
Thank you very much for your suggestion to edit robots.txt and check webmaster tools if any issues. Now, I want to change the default robots.txt but I am not allowed to edit. How can I edit it?
Hello,
Can you please confirm what error you’re getting when you attempt to edit your robots.txt file using Rank Math? The specific error or screenshots of the error would help us in troubleshooting the issue further.
Meanwhile, you should also read this guide on the most common reasons for not being able to edit your robots.txt file using Rank Math.
Thank you for choosing Rank Math
Now, I have updated default robots.txt of rankmath. lets hope it will solved.
Hello,
Sure thing. If you experience the issue again, please do let us know.
Thank you for choosing Rank Math
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.
