Hello,
Thanks for contacting us and sorry for any inconvenience that might have been caused due to that.
This error means that the URL is being blocked by a 4XX error that is not mentioned here: https://support.google.com/webmasters/answer/7440203?hl=en
Can you please share the affected URL so we can check and assist you further?
Looking forward to helping you.
Hello,
I tested the URL with https://httpstatus.io/ and the URLs are retuning 200 (OK) status:
I checked and the URLs are also present in the sitemaps.
While investigating, I noticed that you’ve included multiple sitemap URLs in the robots.txt file and two of them are at the top. Here’s a screenshot: https://i.rankmath.com/zbHMDq
The sitemap URL should always be at the end of the robots.txt and only the main (https://prodator.co.uk/sitemap_index.xml
) sitemap’s URL should be included.
Please remove the other two sitemap URLs. You can follow this guide to edit the robots.txt: https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/
You’re also using the XML sitemaps from Jetpack. It is not recommended to use multiple sitemap plugins as that would not benefit the website in any way and might result in conflicts. If you wish to use the sitemaps of Rank Math, then please disable the Jetpack sitemaps.
Once the above issues are fixed, please clear your website’s cache and Validate fix the issue to see if the issue gets fixed.
Hope this helps.
Thank you.
Hi Prabhat,
Thank you for your response.
As I do not want to make anything worse, could you please have a look at the screenshots and advise me on what should I change?
Text from the screenshot:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourdomain.com/sitemap_index.xml
Regarding the XML sitemaps from Jetpack, I’ve disabled them 🙂
Thank you for your help.
Best
Tomas
Hello,
The only thing that is wrong with your robots.txt is the last line (sitemap link). Please change the URL to your actual website URL. It should look like this:
Sitemap: https://prodator.co.uk/sitemap_index.xml
Once done, validate again your sitemaps and give Google some time to recrawl your site.
I hope that helps.
Thank you.
Hi Reinelle,
Perfect, thank you so much for all your help! Have a good day 🙂
Best
Tomas
Hello,
Glad that helped.
Please feel free to reach out to us again in case you need any other assistance.
We are here to help.
Thank you.