Hello,
We apologize for the inconvenience you’re experiencing with your robots.txt file. It seems that you’ve followed the steps to bypass your caching provider, Cloudflare, as outlined in the articles you’ve mentioned. However, it’s important to note that Rank Math does not create a physical robots.txt file on your site. Instead, the file is created and displayed when a user visits your robots.txt URL.
If your robots.txt file is available at https://rankmath.com/wp-content/uploads/2023/05/Example-of-a-robots.txt-file-1.png you can safely ignore any errors related to the file not being valid. Optionally, you can add the ruleDisallow: /cdn-cgi/` to your robots.txt file and test it again in PageSpeed Insights.
If you’re still encountering issues after following these steps, please feel free to assign this ticket to a human agent using the available buttons, and we’ll be happy to assist you further.
Thank you for bringing this to our attention.
Sources:
Common robots.txt Issues & How to Fix Them » Rank Math
Rank Math Sitemap Issues and Their Fixes » Rank Math
Hello,
Thank you for contacting us and sorry for any inconvenience that might have been caused due to that.
We’ve checked your robots.txt and we can see Rank Math is successfully generating it on /?robots=1. Please check the screenshot in the sensitive data section for reference.
In this case, please check your website’s root directory and see if there is any physical robots.txt file there. If you’re able to find that, please delete it, and after clearing your website cache, check again.
Let us know how it goes. Looking forward to helping you.
Thank you.
I can confirm that there is not a robots.txt file on my server’s root directory. However, when I go to fernwehfilms.com/robots.txt I get the following output:
User-agent: * Disallow: /
I’m unable to index my website because of this output.
Hello,
When we visit https://fernwehfilms.com/robots.txt we see this https://monosnap.com/direct/5n1gkXEgvCNgAoNL0ZHAZHjK4ZNuw5
Also when we go to https://fernwehfilms.com/?robots=1 we see this https://monosnap.com/direct/5n1gkXEgvCNgAoNL0ZHAZHjK4ZNuw5
Since the correct rules appear at https://fernwehfilms.com/?robots=1 it means this is a heavy server level caching issue.
Can you please try clearing your site, plugin, any server-level or cdn level cache, or browser cache to see if that works for you?
You can also contact your server provider or CDN provider to hard reset all server/CDN level caches.
We hope that helps, and please don’t hesitate to get in touch if you have any other questions.
Thank you.
Why doesn’t it work when I access fernwehfilms.com/robots.txt?
I’ve cleared the cache on WordPress, rocket.net, and Cloudflare and it still shows up as:
User-agent: * Disallow: /
I figured it out. Thank you!
Hello,
We are super happy that you have sorted it out.
If you have another question in the future, please feel free to create a new forum topic, and it will be our pleasure to assist you again.
Thank you.
