Robots.txt is different

Hello,

We apologize for any trouble you might be facing due to this issue.

First of all, please make sure that you have the latest versions of Rank Math and your theme and other plugins installed on your website. If not, please update them as soon as possible.

The most likely cause of this issue is a conflict with another plugin or your theme. To find out which one is causing the problem, you need to do some troubleshooting steps. Don’t worry, it’s not hard and it won’t affect your visitors.

Here’s what you need to do:

• Follow this guide to check for plugin conflicts: https://rankmath.com/kb/check-plugin-conflicts/
• Watch this video to see how it’s done: https://www.youtube.com/embed/FcIWRhcUP2c

If the issue is still there:

• Edit your first post on this ticket and add your WordPress and FTP login details in the Sensitive Data section at the bottom. This is a secure area that only our support team can access. You can also use this plugin to create a temporary login link and share it with us: https://wordpress.org/plugins/temporary-login-without-password/
• You can also use this plugin to keep track of any changes we might make on your website (if needed): https://wordpress.org/plugins/wp-security-audit-log/

Before you share any information with us, please make sure you have a full backup of your website. This is very important in case something goes wrong.

We look forward to helping you troubleshoot this issue.

Hello,

The login you have shared is for your staging site, where you can access the robots.txt in this path: /testme/robots.txt

Since it is a staging site, the robots.txt editor is disabled as the search engine visibility for that site is disabled.

Meanwhile, please check your website’s root directory if there’s a robots.txt file. Rank Math’s generated robots.txt file is created dynamically, so it can’t be used to edit a static file from your website files.

Looking forward to helping you.

Thank you.

Hi there,

We appreciate your patience and cooperation in this matter. We have done some troubleshooting and found the cause of the problem that you’re facing.

It turns out that the problem is related to the WP-Optimize plugin that you have installed on your website. WP-Optimize creates a file called wpo-plugins-tables-list.json that “contains a general and anonymous list of plugins and their associated database tables“. This file is not meant to be indexed by search engines, so WP-Optimize automatically adds a Disallow rule to your robots.txt file to prevent that. This Disallow rule looks something like this: Disallow: /wp-content/uploads/wpo-plugins-tables-list.json

This Disallow rule does not affect your ability to edit your robots.txt file as you normally would, but it does mean that WP-Optimize will always add it back whenever you make any changes to your robots.txt file. This is not a harmful or malicious rule, but it might be annoying or confusing for some users. You have a few options to deal with this:

• You could ignore the additional Disallow rule, as this is harmless and does not stop you from adding or editing your robots.txt file – the only thing is whenever you edit your robots.txt file, whatever rule you added or remove, WP Optimize would force the addition of the rule that disallows the indexing of the wpo-plugins-tables-list.json file.
• You could reach out WP Optimize team and speak to them about this, they could have some suggestions

You can learn more about this issue and how other users have dealt with it here:

• https://wordpress.org/support/topic/major-security-issues-with-wpo-plugins-tables-list-json/
• https://support.google.com/webmasters/thread/199899201/robots-txt-wpo-plugins-tables-list-json?hl=en

We hope this helps you understand the situation better and resolve the issue. If you have any other questions or concerns, please feel free to contact us again. We’re always here to assist you.

Thank you for using Rank Math!