robots.txt test has 2 warnings – how can we fix it

#370013
  • Resolved Darius Zak

    2 years, 11 months ago

    Rank Math free

    hi Guys,

    during testing our website myseparateworld.com we found 2 warning related to robots.txt file.
    is this something that your team could help us with? We believe that Rank Math may know the answer to that problem:

    thanks very much for your help!

    Best regards,
    Dariusz

    0 errors 2 warnings
    Line 3
    Allow: /wp-admin/admin-ajax.php

    The standard practice is not to include the Allow order, even though major crawlers such as Google and Bing will include it.

    In a case where both Allow and Disallow clauses are used on the same URL, only the most specific rule will be applied.

    It’s better to err on the side of caution and make your robots.txt code compatible with all crawlers, if you want to have a single file crawled in an otherwise closed off directory, you should first enter in the Allow clause followed by the Disallow.

    This still isn’t the standard, but it is still an accepted practice.
    Line 5

    Sitemap support is not considered “Best Practice”, despite the fact that Google and Bing support it.

Viewing 1 replies (of 1 total)
  • Jaideep Asrani

    2 years, 11 months ago

    Hello,

    Thanks for contacting us and sorry for any inconvenience that might have been caused due to that.

    I have moved the links to the sensitive data section to make them private. And the robots.txt file that our plugin generates, follows the guidelines of Google. If you want to change that then you can edit the file from WP Dashboard > Rank Math > General Settings > Edit Robots.txt.

    Hope this helps. Let us know if you need any other assistance.

    RankBot

    2 years, 10 months ago

    Hello,

    Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

    If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

    Thank you.

Viewing 1 replies (of 1 total)

The ticket ‘robots.txt test has 2 warnings – how can we fix it’ is closed to new replies.