robots.txt

Hello,

We apologize for the issue you are facing with the robots.txt file. The error message "robots.txt is invalid – Lighthouse was unable to download a robots.txt file" can occur for several reasons, even if you are not using a CDN.

Here are a few steps you can take to troubleshoot this issue:

1. Check the Availability of Your robots.txt File: Ensure that your robots.txt file is accessible by visiting `https://rankmath.com/services. If you see the content of your robots.txt file, then it is available.

2. Editing Your robots.txt File: If you need to edit your robots.txt file, you can do so through the Rank Math plugin by navigating to Rank Math SEO → General Settings → Edit robots.txt. If you encounter issues editing it, please refer to our guide on how to edit robots.txt with Rank Math.
   

3. Common Issues: If you still cannot access or edit your robots.txt file, consider the following common reasons:

   • A physical robots.txt file exists in your site’s root folder, which can prevent Rank Math from managing it. You may need to remove this file via FTP or cPanel.
   • Another plugin might be creating the robots.txt file, which would also prevent Rank Math from editing it. You can disable that plugin to resolve the issue.
   • There could be permission issues on your web server that need to be addressed by your hosting provider.

4. Testing Again: After making any changes, test your site again in Lighthouse to see if the error persists.

If you continue to experience issues, please reply to this ticket, and a human agent from our support team will assist you further. Thank you for bringing this concern to our attention!

Sources:
Common robots.txt Issues & How to Fix Them » Rank Math
I Can’t Edit My Website’s Robots.txt » Rank Math

Hello,

Lighthouse requests the actual file as it downloads it, and fails because the file doesn’t exist since Rank Math creates the robots.txt virtually/dynamically.

​​​​​​​To get around this issue, you need to create a physical file on your website, but then you would lose the option to edit it with Rank Math.

Looking forward to helping you.

Hello,

You can ignore the error since it’s a false positive report.

If you want to fix/clear it, then creating the physical file will be your choice.

Looking forward to helping you.