Hello,
We apologize for the issue you are facing with your robots.txt file. The error message you are seeing, "robots.txt is not valid", can occur when using a content delivery network (CDN) like Cloudflare. It’s important to note that Rank Math does not create a physical robots.txt file on your site; instead, it generates the file dynamically when someone visits your robots.txt URL.
To resolve this issue, please ensure that your robots.txt file is accessible at https://yourdomain.com/robots.txt. You can check this by visiting that URL directly in your browser. If it is accessible, you can safely ignore the error in PageSpeed Insights.
Additionally, since you mentioned adding the rule Disallow: /cdn-cgi/, make sure that there are no conflicting rules in your robots.txt file that might be causing the issue. If you have a rule like Disallow: /, it will block all crawlers from accessing your site, which could lead to errors.
You can also use the Robots Testing Tool to check if Google is blocked from crawling specific URLs on your site. This tool can help you identify any issues with your current configuration.
If you’re still having trouble after checking these points, we recommend replying to this ticket with your website URL so that a human agent from our support team can assist you further.
Thank you for raising this concern!
Sources:
Common robots.txt Issues & How to Fix Them » Rank Math
Hello,
It seems like you’ve marked this ticket as resolved.
Can you please confirm if you still need our assistance with this topic or anything else?
Looking forward to hearing back from you.
Thank you.
Hi, no thanks all good, it was actually a new feature from Cloudflare so not related to Rank Math at all.
Hello,
We are glad to know that this issue has been resolved. Thank you for letting us know.
This ticket will be closed now, but you can always open a new one if you have any other questions or concerns. We are here to help you with anything related to Rank Math.
We appreciate your patience and cooperation throughout this process.
Thank you for choosing Rank Math.
