Hello,
I’m sorry to hear that you are facing issues with the changes you made to the robots.txt file via Rank Math General settings not taking effect. If there is no robots.txt file in the root directory, and the changes are not reflecting, there might be a few reasons causing this:
-
Presence of a Physical File: Even if you don’t see a robots.txt file in the root directory, there might be a physical robots.txt file present. Rank Math creates a virtual robots.txt file, but if there is a physical one, it can override the virtual one. You can check for this by logging into your WordPress root directory using cPanel or an FTP application.
-
Redirection Issues: If there are redirection issues, it can prevent the changes from reflecting. In this case, you might need to add specific rewrite rules to the .htaccess file on your site.
-
Plugin Conflict: Another plugin might be creating the robots.txt file, which can conflict with Rank Math. Disabling the plugin responsible for creating the file can help resolve this issue.
If none of these seem to be the cause of the problem, please reply to this ticket, and a human agent from the Rank Math Support team will further investigate and assist you.
Thank you for reaching out, and we appreciate your patience.
Best regards.
Sources:
Common robots.txt Issues & How to Fix Them » Rank Math
I Can’t Edit My Website’s Robots.txt » Rank Math
How to Edit Your Robots.txt With Rank Math SEO » Rank Math
There is no physical file in the root directory. Since that’s the only place (apparently) where it would work, surely it can’t be somewhere else.
I’m not aware of any redirection problems. What should I look for please?
I do not have any other plugins that generate robots.txt.
Do you have any other ideas?
😉
Richard
Hello,
Thank you for contacting us and sorry for any inconvenience that this issue might have caused.
For the dynamically generated robots.txt to work correctly on your site the option to “Discourage search engines from indexing this site” from WordPress Dashboard > Settings > Reading > Search engine visibility should be unchecked.
Kindly check and confirm that this option is not selected.
Don’t hesitate to let us know if you need our assistance with anything else.
Yes. That field is unchecked.
Hello,
In this case, we might need to take a closer look at the settings. Please edit the first post on this ticket and include your WordPress & FTP logins in the designated Sensitive Data section.
Please do take a complete backup of your website before sharing the information with us.
It is completely secure and only our support staff has access to that section. If you want, you can use the below plugin to generate a temporary login URL to your website and share that with us instead:
https://wordpress.org/plugins/temporary-login-without-password/
You can use the above plugin in conjunction with the WP Security Audit Log to monitor what changes our staff might make on your website (if any):
https://wordpress.org/plugins/wp-security-audit-log/
We really look forward to helping you.
Thank you for your involvement here. It is appreciated.
However, I learned that my hosting service was overriding robots.txt and they have now removed their “contribution” and the Rank Math robots.txt now works.
😉
Richard
Hello,
We are glad that you have sorted this out.
If you have another question in the future, please feel free to create a new forum topic, and it will be our pleasure to assist you again.
Thank you.
