Hello,
We apologize for the issue you are facing with the 404 error on your robots.txt file and the missing canonical URLs on your staging website.
Fixing the 404 Error for robots.txt
Rank Math automatically generates a virtual robots.txt file, and if you’re encountering a 404 error, it may indicate that there is a conflicting physical robots.txt file in your site’s root directory. Here’s how to check and resolve this:
-
Check for a Physical robots.txt File:
- Log into your WordPress root directory using cPanel or an FTP application.
- Navigate to FILES → File Manager.
- Look for a file named
robots.txt
. If it exists, delete it.

-
Clear Cache: After deleting the file, make sure to clear your website and server-level cache. You can refer to this guide on how to clear cache in WordPress.
-
Test Again: After clearing the cache, test your robots.txt URL again at `https://rankmath.com/wp-content/uploads/2023/05/robots.txt-is-not-valid-Lighthouse-was-unable-to-download-a-robots.txt-file-error-message.png.
Fixing Missing Canonical URLs
For the missing canonical URLs, ensure that Rank Math is properly configured to generate them:
- Go to Rank Math SEO → General Settings → Links and check if the option to add canonical URLs is enabled.
- If you have custom canonicals set for your posts, they may not appear in the sitemap. Consider removing those custom canonicals to allow Rank Math to generate them correctly.
If these steps do not resolve the issues, please reply to this ticket with any additional details or errors you encounter, and a human agent from our support team will assist you further.
Thank you for raising this concern!
Sources:
Common robots.txt Issues & How to Fix Them » Rank Math
Rank Math Sitemap Issues and Their Fixes » Rank Math
Hello,
In this case, please add the following rewrite rules to your Nginx config:
# START Nginx Rewrites for Rank Math Robots.txt File
rewrite ^/robots.txt$ /?robots=1 last;
rewrite ^/([^/]+?)-robots([0-9]+)?.txt$ /?robots=$1&robots_n=$2 last;
# END Nginx Rewrites for Rank Math Robots.txt File
Please get in touch with your web host for assistance to insert it into your server config properly.
Once done, clear your website and server’s cache and check again the robots.txt URL.
Let us know how this goes.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.