Hello,
We apologize for the issue you are facing with your robots.txt file and indexing settings. To verify if your robots.txt is blocking Google from indexing your pages, you can follow these steps:
- Access Your Robots.txt File: You can typically find it at
yourdomain.com/robots.txt.
- Review the Content: Look for any
Disallow directives that might be preventing Googlebot from crawling specific pages or sections of your site.
- Use the Robots.txt Tester: You can also use the robots.txt Tester to check if specific URLs are blocked. Enter the URL you want to test and select the user-agent (Googlebot) to see if it is blocked or allowed.
If you find that your robots.txt file is indeed blocking access, you can edit it through Rank Math by following these steps:
- Go to Rank Math SEO → General Settings → Edit robots.txt in your WordPress admin area.
- Modify the rules in the code editor to allow access to the necessary pages. Make sure to remove or adjust any
Disallow directives that are blocking the categories and tags you want indexed.
- Click the Save Changes button to apply your modifications.
If you encounter difficulties or want to ensure everything is set up correctly, we recommend considering our professional setup service, which requires a PRO subscription. You can find more information about this service here.
Thank you for raising this concern, and we hope this helps you resolve the issue! If you have further questions or need assistance, feel free to reply to this ticket.
Sources:
How to Fix the "Indexed, though blocked by robots.txt" Error » Rank Math
How to Fix Indexing Issues in Google Search Console » Rank Math
This is an automated reply from Rank Math Support Bot. Your ticket has been assigned to a human agent and we will get back to you shortly.
Hello,
We checked the tag page, and we can confirm it is currently set to index. Please check the screenshot in the sensitive data section for your reference.
In this case, please try to clear your website cache, including any server-level cache, and revalidate the issue on Google Search Console. Once done, give Google some time to recrawl your site.
Let us know how it goes. Looking forward to helping you.
Thank you.
Thank your for your help, just cleared cache also from my end. Will update you with the recrawl updates over the next few days.
Hello,
That is fine.
This ticket will be kept open so you can update us here. However, if it has been closed by the bot, please don’t hesitate to create a new ticket and reference this ticket here.
We look forward to hearing from you.
Hello,
I have updated the sensitive data as requested. Can you please check further?
Thank you.
Okay here’s my update: the tags/category part with the excluded by noindex ones are slowly but surely continue to fall off.
However, it seems my crawled but not indexed ones are still having problems even though I update every few days. Would updating/changing settings with the robots.txt help?
I’ve already got the redirects with deleted stuff and slight difference of URLs with dates going, that’s no problem. Just trying to get going with the next step.
Hello,
We recommend you refer to our guide treating the crawled, currently not indexed error in GSC: https://rankmath.com/kb/crawled-currently-not-indexed/
Don’t hesitate to let us know if you need our assistance with anything else.
Hello,
I have updated the sensitive data as requested. Can you please check further?
Thank you.
I had a look at 1-2 videos including one of yours and it seems most of the other settings on the titles and meta tab seems to have set to noindex and have changed it now to index and also cleared cache.
Hoping this would make the difference.
However, I have a particular issue to discuss and it’s about “no referring sitemaps detected”. What does that mean? What can I do about it? Having looked at inspect URL on a particular post.
Hello,
The “no referring sitemap detected” means the page has been discovered by Google, and Google knows about it, but not through the sitemap.
Please note that a sitemap is only one of many ways Google discovers the URLs of your site. If Google discovers the URL through other means, it may not cross-reference your sitemap to check if the URL is there. Hence the “No referring sitemaps” message.
We hope this helps.
Thank you.
Alright that would mean I would have to change/submit new sitemap.
Also Despite me turning on index settings lately nothing much changed from my end and seems the robots txt i currently have is causing this as well.
How could I edit and update safely on this plugin? As also got the original copy on the file manager that I have with my host.
currently have with the robots txt is not working*
And yet the problem still exists from my end just trying to solve this 1-2 main issues as best possible.
Update: I need a few more days having updated the robots txt file from my end as well as new sitemap. I will then know if I need more questions with assistance from there.
Hello,
Sure, let us know how this goes.
Meanwhile, if you have any other concerns, please don’t hesitate to contact us anytime so we can assist you further.
Looking forward to helping you.
Hello,
I have updated the sensitive data as requested. Can you please check further?
Thank you.
