Hello,
We apologize for the issue you are facing. The error message from Google PageSpeed Insights indicating that "Lighthouse was unable to download a robots.txt file" can occur when using a content delivery network (CDN) like Cloudflare. It’s important to note that Rank Math does not create a physical robots.txt file on your site; instead, it is generated dynamically when someone visits your robots.txt URL.
If your robots.txt file is accessible at `https://rankmath.com/wp-content/uploads/2024/04/image-121.png you can generally ignore this error. You might also consider adding the following rule to your robots.txt file to help with testing:
Disallow: /cdn-cgi/
This can sometimes resolve issues related to CDN configurations.
If you continue to experience problems or have further questions, please reply to this ticket, and a human agent from our support team will assist you further. Thank you for raising this concern!
Sources:
Common robots.txt Issues & How to Fix Them » Rank Math
The Disallow: /cdn-cgi/ comment your AI support suggested is not correct to do.
Maybe you can if the robots.txt is somehow now accessible from googles point of view.
Hello,
This error appears because Rank Math generates the robots.txt file dynamically, meaning there is no physical file in your site’s root directory to download. Instead, the file is created and displayed when someone visits your robots.txt URL.
If the file is accessible at https://yourdomain.com/robots.txt, you can safely ignore this error.
For more details on this error, you can check this URL: https://rankmath.com/kb/fix-common-robots-txt-issues/#robots-txt-is-not-valid-in-pagespeed-insights
Hope that helps and please don’t hesitate to let us know if you have any other questions.
Thank you.
Hello,
We are glad that helped.
Please feel free to reach out to us again in case you need any other assistance.
We are here to help.
Thank you.
