Hello,
Thank you for contacting Rank Math, and sorry for any inconvenience caused.
Please check if there is a robots.txt file in the root directory of your website.
Login to your web server via FTP and delete the robots.txt file present in the root folder.
It is the same folder where you have your wp-content, wp-admin
etc.. folders.
If there is no robots.txt file in the root directory, then please clear the website cache and if you are using any CDN, clear the cache from there as well.
That should fix the issue.
Let us know if the issue persists.
We are here to assist you.
Hi thank you for your reply.
there is no robots.txt on my ftp and i did clean the cache but the issue is still there.
Hello,
We might need to take a closer look at the settings. Please edit the first post on this ticket and include your WordPress & FTP logins in the designated Sensitive Data section.
Please do take a complete backup of your website before sharing the information with us.
It is completely secure and only our support staff has access to that section. If you want, you can use the below plugin to generate a temporary login URL to your website and share that with us instead:
https://wordpress.org/plugins/temporary-login-without-password/
You can use the above plugin in conjunction with the WP Security Audit Log to monitor what changes our staff might make on your website (if any):
https://wordpress.org/plugins/wp-security-audit-log/
We really look forward to helping you.
Hello,
I have updated the sensitive data as requested. Can you please check further?
Thank you.
Hello,
I checked your robots.txt file and upon investigating, found that as a user logs in to the website, the sitemap statement moves to a new line and upon logging out, it comes to the same line. You can have a look at the screencast in the sensitive data section of this ticket.
This usually happens due to heavy caching. If you have cleared the website’s cache, please get in touch with your web host and see if they’re caching the file from their end.
You can also create a staging website and one by one disable the plugins and the theme manually and at the same time check the robots.txt file in an incognito window to see if there’s a plugin/theme causing this.
Here’s a guide that you can follow to create the staging website: https://www.wpbeginner.com/wp-tutorials/how-to-create-staging-environment-for-a-wordpress-site/
Let us know how that goes.
Thank you.
Hi the solution i was given by my hosting provider is to place a physical robots.txt file in the ftp. In fact now it does work as it should do.
Would having a physical robots.txt file cause any issue with rankmath and SEO?
Hello,
No, having a physical robots.txt file would not cause any issues with your SEO. The only downside would be that you’ll not be able to edit that file with Rank Math and you’ll have to edit it manually.
If you do not frequently edit the robots.txt, then you can easily go ahead with the physical robots.txt file.
Hope this helps.
Thank you.
Hello,
We’re glad that we could address the concern.
Please feel free to get in touch if you require any further assistance.
Thank you.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.
