URL is not available to Google

#130106
  • Resolved Lee
    Rank Math free

    Hi, my sitemap shows an 404 error. Can ypu check why there is a noindex’ detected in ‘X-Robots-Tag’ http header? Thank you

Viewing 15 replies - 1 through 15 (of 15 total)
  • Danial
    Rank Math business

    Hello,

    Thank you for contacting Rank Math and sorry for any inconvenience that might have been caused due to that.

    Could you please do the following:
    1. Head over to WordPress Dashboard > Settings > Permalinks
    Hit the save button once.

    2. Go to WordPress Dashboard > Rank Math > Sitemap Settings
    Hit the save button once.

    You can refer to this link from our KB:
    https://rankmath.com/kb/fix-sitemap-issues/#404-error

    I hope this resolves your issue.

    Lee
    Rank Math free

    I have done as instructed above but still face the same issues:

    Sitemap can be read, but has errors

    Hello,

    We see that it is working fine: https://thesmartbackyard.com/sitemap_index.xml

    NoIndex is normal on a Sitemap URL and nothing to be worried about.

    A Sitemap is set to noindex so the sitemap itself does not appear in the search results.

    Hope that helps and please do not hesitate to let us know if you need our assistance with anything else.

    Lee
    Rank Math free

    allow me to share 3 screenshots maybe I will be understood better. My sitemap displays an error and with it is a tanking pageviews from google from 1k sessions to only 500 as of yesterday! This always happens everytime the sitemap error occurs.

    Do you mind checking it out in person and troubleshoot where the problem might be? thank you

    Hello,

    When I checked your sitemap from my end, it was redirecting to homepage. I logged into your site and changed the links per sitemap in Rank math> Sitemap Settings to 201 from 2 after which the sitemap started loading. This suggests that your sitemap is still being cached and the issue is likely to recur.

    Since you have breeze installed and have excluded your sitemaps from being cached internally, to exclude URLs from the Varnish cache, please refer to this article: https://support.cloudways.com/how-to-exclude-url-from-varnish/

    If you also have Cloudflare, please follow this article: https://rankmath.com/kb/exclude-sitemaps-from-caching/

    Once you are done with all this, you should remove any sitemap you have submitted on your GSC and resubmit only the main sitemap: /sitemap_index.xml

    That should fix the issue. Please let me know if that does not. We are here to assist.

    Lee
    Rank Math free

    Hello,

    I have updated the sensitive data as requested. Can you please check further?

    Thank you.

    Lee
    Rank Math free

    I have followed all the steps as instructed above, including varnish settings rule, cleared cache and still the problem is not solved.

    Can we try another way to see if it can be solved? Thank you

    Note: I have added a screenshot with the GSC issue. It says no index detected at HTTP
    header

    Thank you

    Danial
    Rank Math business

    Hello,

    Have you remove any sitemap you have submitted on your GSC and resubmit only the main sitemap?

    Please let us know.

    I hope this resolves your issue.

    Lee
    Rank Math free

    Yes i did. I followed every step above. I would request someone from your team to troubleshoot the issue. It’s taking days for it to be resolved and a single issue like this taking too long will only overwhelm your customer care with queries. Thank you

    Hello,

    Sorry for the delay.

    Based on your added screenshot, it is completely normal to have noindex on the sitemap URLs. It is done on purpose, so your sitemaps do not become part of the index.

    Since you have already submitted your sitemap_index.xml file to Google, please give it some time to recrawl your website.

    I hope that helps. Thank you, and please don’t hesitate to contact us if you need further assistance.

    Lee
    Rank Math free

    Thank you

    Miya
    Rank Math free

    Hi lee,

    I too have same issue, have you had it fixed yet? If yes, please let me know your solution. Thanks in advance.

    Lee
    Rank Math free

    @Miya I haven’t found a solution yet. I still have a 404 error on my sitemap 7 days later. I was asked to wait for a few days, it’s one week and I still have the same error. I don’t know if the plugin is malfunctioning, conflicting with another plugin or something on my site or a bug developers at RankMath haven’t discovered yet?

    If it takes more than 12 days, I will be forced to try another plugin and see if the frequent sitemap errors will still occur. Migrating feels cubersome though and i dont feel like going back to Yoast given how robust RankMath is in terms of functionality and features.

    The funny part is the sitemap map was scrawled yesterday and it shows 14 categories! when I have 4 categories and have never gone beyond 6 categories…haha… I also see that I have 80 posts last crawled 2 days ago yet I have 124 published posts. I dont know where this data comes from.

    I am sure there is a problem with the Plugin they haven’t discovered yet.

    Miya
    Rank Math free

    @Lee,

    I have added my sitemap in Google Search Console including 13 pages , 59posts, 35 category. totally 107 links. I have no 404 error, but it seems google couldn’t fetch my sitemap.

    The sitemap status is success, but when I use URL inspection tool, just few URLs are on Google and even more strange is that coverage is shown “Indexed, not submitted in sitemap”.

    Th big problem is that majority of the URLs has issue “is unknown to Google”, I have no ideia how to make google to crawl these urls and index.

    I need to do a lot of search to find a solution, once I figure out, I will send you message. Thanks.

    Hello,

    Are the URLs marked as Indexed, not submitted in sitemap in your sitemap ? Could you please share some of these URLs in the sensitive section so we can check from our end?

    Looking forward to helping you.

Viewing 15 replies - 1 through 15 (of 15 total)

You must be logged in to reply to this ticket.