Still getting robots.txt is not valid by google after adding Disallow: /cdn-cgi/

Hello,

We’ve looked into your site and your robots.txt file and we can confirm that they are both fine and working as expected. Your robots.txt file is valid and readable by both humans and bots, and it follows the best practices for SEO.

However, please note that since you are using Rank Math for your robots.txt file, we are actually not creating a physical file on your server but a virtual one that is generated dynamically by Rank Math. This is a perfectly acceptable method by Google and other search engines, but it might cause some issues with some applications that try to download the actual file from your server. In that case, they will get an error because the file does not exist physically.

If you want to avoid this error, you have two options:

• You can create a physical robots.txt file on your server using FTP or cPanel. You can use the same content that you have in Rank Math, or you can customize it as you wish. However, if you do this, you will lose the ability to edit your robots.txt file with Rank Math. You will have to edit it manually every time you want to make changes.
• You can ignore the error and continue using Rank Math to edit your robots.txt file. This will not affect your SEO or your site performance, as long as your robots.txt file is accessible by search engines. You can check this by using the robots.txt tester tool in Google Search Console or other online tools that do not require downloading the actual robots.txt file.

Either way, we recommend that you follow the rules that we suggest for the robots.txt file in this article: https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/#default-rules

We hope this helps and clears up any confusion. If you have any questions or concerns, please let us know. We’re always here to help.

Thank you for choosing Rank Math.