Hello,
Thank you for contacting Rank Math today.
I have checked it and the page seems fine, it has no no-index set so it should index without issues.
But I found some issues in your robots.txt file. You don’t need the: Allow: /sitemap.xml since Rank Math doesn’t add that URL (it uses sitemap_index.xml) and in fact you don’t want (never) your XML sitemaps indexed, so in any case you wouldn’t set it as Allow.
But make sure to include these lanes in your robots.txt file (changing that domain name with yours):
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: http://mywebsite.com/sitemap_index.xml
Looking forward to helping you. Thank you.
Thanks that URL error is solved.
But as you said that sitemap shouldn’t be indexed then why we allowed that sitemap_index.xml?
And, What’s that Admin-ajax.php?
Hello,
We used the Sitemap: rule, not the Allow: one. It is used to let search engines know where is the sitemap (if they don’t have it using other techniques like adding it in Search Console).
This file is often used front-end by themes and plugins. It was changed in WordPress robots.txt file a couple of years ago to allow crawling for this reason, after an issue with google being unable to crawl it.
The WP Core ticket thread can be found here and offers further information https://core.trac.wordpress.org/ticket/33156
Looking forward to help you.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.
