Robot.txt is invalid?

#579731
  • Resolved JR
    Rank Math free

    Hi,

    The Devtool tells that my Robot.txt is not valid. Do you know how to solve the issue?

Viewing 3 replies - 1 through 3 (of 3 total)
  • Hello,

    Thank you for reaching out and using Rank Math.

    I have checked your site and your robots.txt file, and I didn’t find any problem with them. Your robots.txt file is valid and accessible by humans and bots alike. I don’t know why PageSpeed Insights would have trouble downloading your robots.txt file, but this is not your fault.

    However, you can try adding this line to your robots.txt file using Rank Math’s Robots.txt Editor. You can find it by going to Rank Math > General Settings > Edit Robots.txt.

    Disallow: /cdn-cgi/

    Some users have reported that adding this line helped.

    After adding this line, please clear your site’s cache and test your site again with PageSpeed Insights. See if this makes any difference.

    I hope this helps you to fix the issue with PageSpeed Insights. If you have any more questions or doubts, please feel free to ask me. I’m here to help you.

    Thank you for choosing Rank Math.

    JR
    Rank Math free

    Hi,

    Thanks for your reply. The thing is I’m not using CloudFlare. Is there anything else I could do?

    Hello,

    If you are using our plugin to modify this file the error is normal since we don’t actually create a physical file on your website but create a virtual one.

    This method is completely acceptable by Google but if an application requests the actual file download it will fail because the file doesn’t exist.

    ​​​​​​​To get around this issue you need to create a physical file on your website but then you would lose the option to edit it with Rank Math.

    Either way is fine, as long as you’re also using the rules we are recommending for the robots.txt:
    https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/#default-rules

    Hope this helps clarify your doubts.

    Don’t hesitate to get in touch if you have any other questions.

    Hello,

    Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

    If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

    Thank you.

Viewing 3 replies - 1 through 3 (of 3 total)

The ticket ‘Robot.txt is invalid?’ is closed to new replies.