Hello,
We’re sorry to hear about the issue you’re facing with the URL not being indexed. It seems like Googlebot is being blocked by the robots.txt file, which could be the reason for the non-indexing.
To resolve this problem, you can use the Robots Testing Tool to check if Google is blocked from crawling the specific URL on your site. If the tool doesn’t give you any error, it’s possible that there might be a discrepancy between the robots.txt file displayed in the tool and your current robots.txt file.
You can also use Rank Math’s Analytics to identify pages with the problem. Navigate to Rank Math SEO → Analytics in the WordPress dashboard and click on the Index Status tab. Under this tab, you’ll get the real data/status of your pages as well as their presence on Google.
If you need further assistance, please feel free to contact our dedicated support team. We’re available 24/7, 365 days a year and we’ll be happy to help you troubleshoot and fix this issue.
We hope this helps!
Sources:
No Information is Available for This Page » Rank Math
How to Fix ‘Blocked by robots.txt’ Error in Google Search Console » Rank Math
Common robots.txt Issues & How to Fix Them » Rank Math
Why GSC Shows “No Referring Sitemaps Detected”
Hello,
Thank you for your query, and we are so sorry about the trouble this must have caused.
It sounds like you are encountering a common issue with Google Search Console (GSC) and the indexing of a particular URL on your website.
The error message “Googlebot is blocked by the robots.txt file” is indeed a significant clue to why your URL is not being indexed.
We checked the robots rules at https://emmanuelle-wiesemes.com/robots.txt .
The user-agent Google-Extended is not a standard one recognized by Google.
The typical user-agents used by Google are “Googlebot” for their main web crawler and “Googlebot-Image” for image crawling.
Disallow: /: This is a major issue. If “Google-Extended” is interpreted by Google as a valid user-agent, this rule blocks Google from crawling your entire site.
Basically if Googlebot respects this directive, it will not crawl any part of your site.
Since “Google-Extended” is not a standard user-agent, it’s unclear how Googlebot interprets this directive.
Also the rule Disallow: /wp-content/ is potentially problematic as it can block important content from being crawled.
To be safe, you should remove it.
Here’s a guide on how to edit your robots.txt using Rank Math:
https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/
We hope that helps, and please don’t hesitate to get in touch if you have any other questions.
Thank you.
HI Alhan
Currently I can’t edit the robots txt in Rankmath. I get the message “Content is locked because a robots.txt file is present in the root folder.”
What should I do now to let Rankmath (re)generate it by default?
Thanks
Hello,
If you are not able to edit the robots.txt file, please follow this guide: https://rankmath.com/kb/cant-edit-robots-txt/
Let us know how that goes. Looking forward to helping you.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.
