Hello,
Thank you for contacting Rank Math and sorry for any inconvenience that might have been caused due to that.
Your robots.txt seems to be only accessible with a slash at the end: https://alansflowersandgifts.com/robots.txt/
Could you please try creating a robots.txt file on your root folder and check if this is the same. If so it is likely there is some misconfiguration in your hosting settings or in your .htaccess file. Could you let us know once your hosting support check it?
Looking forward to helping you. Thank you.
Hello Michael,
Thank you for the response! With a real file, /robots.txt works. However, at least at first, /robots.txt/ redirects to a different site. As for the .htaccess, the server is using nginx, but I could share server config in the sensitive data container.
Hi,
Since a robots.txt file already exists, you need to remove that so that the one by Rank Math will take effect.
If that doesn’t work, we might need to take a closer look at the settings. Please include your WP logins in the designated Sensitive Data section.
It is completely secure and only our support staff has access to that section. If you want, you can use the below plugin to generate a temporary login URL to your website and share that with us instead:
https://wordpress.org/plugins/temporary-login-without-password/
You can use the above plugin in conjunction with the WP Security Audit Log to monitor what changes our staff might make on your website (if any):
https://wordpress.org/plugins/wp-security-audit-log/
We really look forward to helping you.
Hello,
I have updated the sensitive data as requested. Can you please check further?
Thank you.
Hello,
The robots.txt file was only created to test Michale’s question about how our server configuration treated an actual .txt file. As it stands the virtual can only be accessed with a slash at the end, while a real one can be accessed without it. I have now deleted the real txt file so it won’t cause conflicts, though this did not fix the issue. I too hope that whatever error is causing this problem can be found quickly and easily.
Hello,
I couldn’t login to your site with the username/password you added in the Sensitive Data Section. Can you please check and add the correct details?
Looking forward to helping you.
Hello,
I have updated the sensitive data as requested. Can you please check further?
Thank you.
I updated the sensitive info, and checked that this password works.
Hello,
The robots.txt file is showing 404 on your site is because your server is still looking for the physical file in your root directory when the link is without the slash. Please try adding following code in your Nginx config file:
location = /robots.txt {
try_files $uri $uri/ /index.php?$args;
access_log off;
log_not_found off;
}
If the page still shows 404 even after adding this code then please get in touch with your hosting team and ask them if they have any internal rules set in the server to prevent the dynamic links without slash from executing.
I hope that helps.
That works! I realized my server configuration had a line for robots.txt. I removed it, and replaced it with that, and all’s good!
Hello Geoff,
We are super happy that this resolved your issue. If you have any other questions in the future, know that we are here to help you.
If you don’t mind me asking, could you please leave us a review (if you haven’t already) on https://wordpress.org/support/plugin/seo-by-rank-math/reviews/#new-post about your overall experience with Rank Math? We appreciate your time and patience.
If you do have another question in the future, please feel free to create a new forum topic, and it will be our pleasure to assist you again.
Thank you.
