Hello,
We apologize for the issue you are facing with the invalid robots.txt file. It seems like there might be a few reasons why this is happening. Since you mentioned that no physical robots.txt file exists, let’s explore some potential solutions:
-
Check for Conflicting Plugins: Ensure that no other plugins are creating a robots.txt file. If another SEO plugin is still active, it might conflict with Rank Math.
-
Clear Cache: If you are using any caching plugins or a CDN, please clear your website and server-level cache. This can sometimes resolve issues with how files are served.
-
Check Rewrite Rules: If your site runs on an Nginx server, ensure that the appropriate rewrite rules are added to your server configuration. For Apache servers, check your .htaccess file for any necessary rules.
-
Test the URL: Make sure that you can access your robots.txt file directly by visiting `https://rankmath.com/blog/clear-cache-in-wordpress/. If it returns a 404 error or a blank page, it indicates that there might be a redirection issue or a misconfiguration.
-
PageSpeed Insights Error: If you are seeing an error in PageSpeed Insights stating that the robots.txt is not valid, you can safely ignore this if your robots.txt file is accessible at the correct URL. You may also add the following rule to your robots.txt file to help with CDN-related issues:
Disallow: /cdn-cgi/
If these steps do not resolve the issue, please reply to this ticket with any additional details or errors you are encountering, and a human agent from our support team will assist you further. Thank you for bringing this concern to our attention!
Sources:
Common robots.txt Issues & How to Fix Them » Rank Math
None of these points are helpful. There is no other robots.txt file. The actual robots.txt
url is invalid: https://www.xml-sitemaps.com/validate-xml-sitemap.html.
I cannot find info for how to generate a valid robots.txt in Rank math.
Thank you
Kerry
Hello,
Thank you for contacting us and sorry for any inconvenience that this issue might have caused.
Assuming you are using the latest version of all the plugins and the themes (please update if not yet), it could be caused by a conflict with one of your plugins/theme.
To determine that, please follow this troubleshooting guide:
https://rankmath.com/kb/check-plugin-conflicts/
Only the site admin would know and your users won’t be affected by this troubleshooting.
Let us know how this goes.
Thank you.
Hi Ike
Thank you for your reply.
Once all my plugins were disabled, and only Rankmath enabled, the sitemap worked. I then enabled the plugins one by one and when I got to the last one, the sitemap was still working!
I verified it on the Rankmath verification tool and uploaded it to Console and it was fine.
Very confusing.
Thank you for your help. Much appreciated.
Kerry
Hello,
We’re delighted to hear that this issue has been resolved. We appreciate your feedback.
This ticket will be closed now, but if you ever have another question or need any help in the future, please don’t hesitate to create a new forum topic. We’ll be more than happy to assist you again.
Thank you.
