Hello,
Thank you for contacting Rank Math and bringing your concern to our attention. I’m sorry for any inconvenience this issue may have caused you.
I’ve checked your site, and it seems that you only have 3 pages. However, your sitemap is already working fine.
Upon investigating, it seems that you have a static robots.txt file in your website root directory as we’re getting Contents are locked because a robots.txt file is present in the root folder message in Rank Math > General Settings > Edit robots.txt
To let Rank Math generate the recommended robots.txt on your site, please delete (or at least rename) the file on your website. You can get in touch with your web host for assistance.
Here’s a link for more information:
https://rankmath.com/kb/cant-edit-robots-txt/#num-1-presence-of-a-physical-file
I hope that helps.
Thank you.
So, you want us to delete the robots.txt file which is placed in our site’s root directory? Please confirm. Thanks.
We have deleted the robosts.txt file from the root. Can you please check now & confirm? Thanks.
Hello,
I’ve checked your /robots.txt, and it is already showing the recommended rules for your site to get indexed.
In this case, please remove all the sitemaps submitted in your Google Search Console account and resubmit your main sitemap only /sitemap_index.xml.
Here’s a link for more information:
https://rankmath.com/kb/submit-sitemap-to-google/
Once done, give Google some time to recrawl your site.
I hope that helps.
Thank you.
We have resubmitted the https://www.vestproduct.in/sitemap_index.xml in GSC. Later we tried to inspect the sitemap URL but getting below errors in search console –
Time – Jan 27, 2022, 10:44:35 AM
Crawled as – Googlebot smartphone
Crawl allowed? – Yes
Page fetch – Successful
Indexing allowed? – error
No: ‘noindex’ detected in ‘X-Robots-Tag’ http header
What to do here?
Hello,
If you’re getting that error for the sitemap URL, then there’s nothing to be worried about as the sitemaps are by default set as noindex. We do this as sitemaps themselves should not be indexed on Google. Also, keeping the sitemaps as noindex does not, in any manner, blocks the bots from crawling them.
I also tested the variations of your website with https://httpstatus.io/, and two variations are returning the 200 status.
Except for the preferred variation (which should return only a 200 status), all the non-preferred variations of the website should redirect to the preferred variation and return a 301 + 200 status.
Please follow the steps in this link:
https://rankmath.com/kb/couldnt-fetch-error-google-search-console/#5-check-for-the-correct-version-of-the-sitemap and redirect to the correct variation.
You can also get in touch with your web host regarding this.
Once this is fixed, please exclude the sitemaps from being cached. Here’s a guide that you can follow: https://rankmath.com/kb/exclude-sitemaps-from-caching/
Then, clear your website’s cache and remove all the sitemaps from your Google Search Console account, and then submit only the main sitemap: /sitemap_index.xml.
Further, give Google some time to crawl your website again and see if the contents start getting indexed.
Hope this helps.
Thank you.
Thanks for the clarification.
Well to check the correct version of sitemap site:mysite.in is not giving any result and that is our main concern & we raised this ticket. However, it is redirecting from non-www/http to www/https automatically. Hence confused what to do here.
For the second point as of now we are not using any cache plugin so only .htaccess hack is left as per this article (https://rankmath.com/kb/exclude-sitemaps-from-caching/). Should we go ahead and ad the below line in .htaccess file?
RewriteEngine On
RewriteRule .* – [E=Cache-Control:no-cache]
Please help. Thanks.
Any updates on this please?
Hello,
It seems like you have a firewall rule preventing access from Googlebot on your website and that could be the root cause of this:
Please make sure that you remove any blockage on Googlebot from your website so it can be crawled and possibly indexed.
Don’t hesitate to get in touch if you have any other questions.
OMG!!.. we haven’t added any firewall explicitly however we are using CloudFlare CDN but we haven’t enable such there. Can you please let me know like how can we remove this or who could be the culprit behind this?
Thanks.
Any updates on this please? Thanks.
Hello,
You can get in touch with Cloudflare about this situation and ask them to have a closer look at the situation.
If the blockage is not happening from their side you need to get in touch with your hosting provider and report the same so that the blockage can be lifted and Googlebot can finally access your website.
Don’t hesitate to get in touch if you have any other questions.
Thanks Miguel.
Well we have just now deleted the cloudflare. Can you please now check now whether any firewall rule still preventing access from Googlebot?
Please let us know your findings. Thanks.
Hello,
I can check that Googlebot is still getting blocked on your site:
Please get in touch with your host and inform them about the issue as they will be able to help you investigate this further.
Let us know how that goes.
Thanks.
We checked with hosting & they said that it may be blocking due to one security layer added by them for xmlrpc attacks. So they disabled it temporarily.
Can you please check again & let me know? Thanks.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.
