Not Indexing, Blocked by Robots.txt

#25210
  • Resolved Chad
    Rank Math pro
    Content AI Starter

    I’ve noticed that lots of products are no longer showing correctly on search results.

    I tested in URL inspection on Google Search Console and it comes back as crawl not allowed and page fetch blocked by robots.txt.

    There is no robots.txt file in my my base directory. I haven’t made any changes to settings in Rank Math.

Viewing 2 replies - 1 through 2 (of 2 total)
  • Todd
    Rank Math free

    Hello Chad,

    Thanks a lot for your query and we are so sorry about the trouble this must have caused.

    Can you please:
    1. Link us to your robots.txt file
    2. Share a list of what URLs aren’t showing correctly
    3. Tell us what you expect to see instead of what you are seeing

    Please provide this information so we are in a better position to help you.

    Thank you.

    Sorry for not posting more useful information. So this is what caused me to look into this. I was just doing a simple google search for products I know we carry and rank high.

    https://prnt.sc/qhxp5a

    As you can see one page wasn’t indexed, the other isn’t showing rich snippet info (price or availability.

    I don’t have a robots.txt file, I checked my root directory and it does not exist.

    This is what is shown using the search console robots.txt tester:

    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php
    
    Sitemap: https://www.moreheadspeedworks.com/sitemap_index.xml

    But last night it wasn’t showing that sitemap link. The robot.txt tester always showed as allowed, but when I used the URL inspector in Search Console, it would show nearly all links were blocked by robots.txt.

    This is my website: https://www.moreheadspeedworks.com/

    Hello,

    Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

    If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

    Thank you.

Viewing 2 replies - 1 through 2 (of 2 total)

The ticket ‘Not Indexing, Blocked by Robots.txt’ is closed to new replies.