Hello,
Thanks for contacting us and we regret the inconvenience caused.
Could you please confirm if you have a physical robots.txt file present in the root folder of your website? If yes, then the changes made in the Robots.txt option of Rank Math wouldn’t work until the physical file is deleted.
For more details, please refer to this tutorial guide: https://rankmath.com/kb/cant-edit-robots-txt/
Please let us know how that goes.
Thanks.
Hello,
Thank you for the response.
No, I have no physical robots.txt in my WP. In fact, if I change the content of the Rank Math robots.txt switching the site to live, it takes the changes: https://pruebacurso.lucusprueba.es/robots.txt
The problem exists when the site is blocking bots from Settings > Reading and the robots.txt has the default config from Rank Math: http://prntscr.com/11d6jol
Thank you.
Hello,
Thanks for your reply.
Since the Robots.txt file is mainly for the search engines, you would not be able to edit the Robots.txt file if you’ve enabled the “Discourage search engines from indexing this site” option from your WordPress admin area > Settings > Reading.
A warning for the same would also be displayed below the Robots.txt editor in your WordPress admin area > Rank Math > General Settings > Edit robots.txt.
Also, if you edit the robots.txt and then enable the “Discourage search engines from indexing this site” option, then the contents of the Robots.txt file would be changed to default.
Hope this helps.
Thanks.
Hello,
When you say this:
“Also, if you edit the robots.txt and then enable the“ Discourage search engines from indexing this site ”option, then the contents of the Robots.txt file would be changed to default.”
… the result is this: http://prntscr.com/122qyvx
Do you confirm me that it is enough with enable the option “Discourage search engines from indexing this site” to avoid robots to index the site although the robots.txt shows that configuration?
I have read in several technical WP blogs and forums that the robots.txt should be like this although “Discourage search engines from indexing this site” be enable:
User-agent: *
Disallow: /
Thank you.
Hello,
Thanks for getting back to us.
Changing the robots.txt to default when “Discourage search engines from indexing this site” is enabled, is the intended behavior. We do that to make sure that there is nothing in the robots.txt file conflicting with the setting to discourage the search engines.
Even if the robots.txt allows bots to crawl the website, if there’s a No Index tag applied, then the bots will see it and should not Index it.
Hope this clears things further.
Thanks.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.
