robot txt errors on fastcasualstorage.com

#345328
  • Resolved Justin Rule

    3 years, 10 months ago

    Rank Math free

    Why is the sitemap URL not included in the Robots.txt file?
    What is the purpose of line 7 in the robots.txt file? This appears to be blocking all the pages our plugin creates, so I’m curious if there is some purpose to this that I’m not seeing.

    The robots file is it appears to be blocking Google from crawling the pages we created.

    I would recommend removing that disallow directive.

    You can see why this is problematic by going to Google and entering “site:fastcasualstorage.com”. There are only 6 pages indexed on Google for the domain, which matches up with the robots file blocking the additional pages.

    This is clearly problematic.

Viewing 1 replies (of 1 total)
  • Anas

    3 years, 10 months ago

    Rank Math business

    Hello,

    Thank you for contacting Rank Math, and sorry for any inconvenience caused.

    The Disallow: /*? is used to block access to all URLs that include a question mark (?). More specifically, any URL that begins with your domain name, followed by any string, followed by a question mark, followed by any string.

    You can add the sitemap URL in the robots.txt by editing the robots.txt file.

    You can edit the robots.txt with Rank Math and add the sitemap URL.
    https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/

    If you are not sure about the directives in the robots.txt, you can use the default robots.txt file:

    User-Agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourdomain.com/sitemap_index.xml

    Hope that helps.

    Looking forward to helping you.

    RankBot

    3 years, 9 months ago

    Hello,

    Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

    If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

    Thank you.

Viewing 1 replies (of 1 total)

The ticket ‘robot txt errors on fastcasualstorage.com’ is closed to new replies.