Sitemap blocked by robots.txt

#495792
  • Resolved aleexxjohansson
    Rank Math free

    Hi!

    I am having issues with submitting my sitemap to google search console. It says that it can’t be read and if I click on it it says HTTP error. I tried submitted the sitemap url into the submit url in google search console and there it stood that it was blocked by robots.txt.

    But I haven’t made any changes to the robots.txt file. Can you take a look at this?

    //Alex

Viewing 5 replies - 1 through 5 (of 5 total)
  • Hello,

    Thank you for contacting Rank Math support, and sorry for any inconvenience that might have been caused due to that.

    We checked your sitemap and it seems to be loading just fine.

    Your website’s robots.txt should also allow your sitemap to be crawled by the bot as you can see in the screenshot below:

    test

    The sitemap error “Couldn’t Fetch” that you might come across is most likely a bug with Google Search Console. To troubleshoot more on this, please follow this detailed guide: https://rankmath.com/kb/couldnt-fetch-error-google-search-console/#num-1-configure-your-sitemap.

    Looking forward to helping you with this one.

    I followed the guide and on https://httpstatus.io/ I got two with only 200 code but I now redirected to only use https://norrlandsfritid.se/sitemap_index.xml but it still doesn’t work.

    Hello,

    I can see that your sitemap is no longer being blocked by robots.txt. Here’s a screenshot below for your reference:

    Please try removing the existing sitemap from GSC and re-submit the /sitemap_index.xml there to see if that fixes the issue for you.

    Looking forward to helping you.

    Thank you.

    It worked now but only the custom post type I have not the regular pages.
    Image: https://gyazo.com/8226676e3a4d6a2aebb2ce1b98787fb5

    Hello,

    We have doubled checked your sitemap again on our end and we can confirm the sitemap is loading just fine as you can see in the screenshots below:

    sitemap

    To further address the issue with GSC, please try following the steps below:

    1. Flush the Sitemap cache by following this video screencast:
    https://i.rankmath.com/pipRDp

    2. Exclude the Sitemap files of the Rank Math plugin in your caching plugin. The cache could be via a plugin or from the server. For plugins or Cloudflare, please follow this article:
    https://rankmath.com/kb/exclude-sitemaps-from-caching/

    3. Add the filter code to your theme’s functions.php file:

    add_filter( 'rank_math/sitemap/enable_caching', '__return_false');

    4. Remove all your sitemaps again from your Google Search Console and resubmit only the primary one (sitemap_index.xml).

    Looking forward to helping you with this one.

    Hello,

    Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

    If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

    Thank you.

Viewing 5 replies - 1 through 5 (of 5 total)

The ticket ‘Sitemap blocked by robots.txt’ is closed to new replies.