My robots.txt URL Returns a 404 Error

Hello,

We apologize for the issue you are facing with your robots.txt file returning a 404 error. Since you have deleted the existing robots.txt file, it may indicate that there is a conflict with another robots.txt file or a redirection issue.

Here are the steps you can take to resolve this:

1. Check for Another robots.txt File: Ensure that there are no other robots.txt files in your site’s root directory. You can do this by logging into your WordPress root directory using cPanel or an FTP application. If you find another robots.txt file, please delete it.

   

2. Clear Cache: After ensuring there are no conflicting files, clear your website and server-level cache. This can help in reflecting the changes made.

3. Add Rewrite Rules: If your root directory does not contain a robots.txt file, you may need to add the applicable rewrite rule to your .htaccess file. Here’s how:

   • For Apache servers, add the following rule:

     RewriteRule ^robots.txt$ /index.php?robots=1 [L]

   • For Nginx servers, use:

     # START Nginx Rewrites for Rank Math Robots.txt File
     rewrite ^/robots.txt$ /?robots=1 last;
     rewrite ^/([^/]+?)-robots([0-9]+)?.txt$ /?robots=$1&robots_n=$2 last;
     # END Nginx Rewrites for Rank Math Robots.txt File

4. Test Your robots.txt URL: After making these changes, visit https://yourdomain.com/robots.txt to see if it loads correctly.

If you continue to experience issues, please reply to this ticket, and a human agent from the Rank Math Support team will assist you further.

Thank you for raising this concern!

Sources:
Common robots.txt Issues & How to Fix Them » Rank Math