robots.txt throwing 404 error and Canonical URL is also missing

#1034932
Viewing 2 replies - 1 through 2 (of 2 total)
  • RankBot

    3 months, 1 week ago

    Hello,

    We apologize for the issue you are facing with the 404 error on your robots.txt file and the missing canonical URLs on your staging website.

    Fixing the 404 Error for robots.txt

    Rank Math automatically generates a virtual robots.txt file, and if you’re encountering a 404 error, it may indicate that there is a conflicting physical robots.txt file in your site’s root directory. Here’s how to check and resolve this:

    1. Check for a Physical robots.txt File:

      • Log into your WordPress root directory using cPanel or an FTP application.
      • Navigate to FILES → File Manager.
      • Look for a file named robots.txt. If it exists, delete it.
        Delete robots.txt in your WordPress root directory

    2. Clear Cache: After deleting the file, make sure to clear your website and server-level cache. You can refer to this guide on how to clear cache in WordPress.

    3. Test Again: After clearing the cache, test your robots.txt URL again at `https://rankmath.com/wp-content/uploads/2023/05/robots.txt-is-not-valid-Lighthouse-was-unable-to-download-a-robots.txt-file-error-message.png.

    Fixing Missing Canonical URLs

    For the missing canonical URLs, ensure that Rank Math is properly configured to generate them:

    • Go to Rank Math SEO → General Settings → Links and check if the option to add canonical URLs is enabled.
    • If you have custom canonicals set for your posts, they may not appear in the sitemap. Consider removing those custom canonicals to allow Rank Math to generate them correctly.

    If these steps do not resolve the issues, please reply to this ticket with any additional details or errors you encounter, and a human agent from our support team will assist you further.

    Thank you for raising this concern!

    Sources:
    Common robots.txt Issues & How to Fix Them » Rank Math
    Rank Math Sitemap Issues and Their Fixes » Rank Math

    Reinelle

    3 months, 1 week ago

    Hello,

    In this case, please add the following rewrite rules to your Nginx config:

    # START Nginx Rewrites for Rank Math Robots.txt File
rewrite ^/robots.txt$ /?robots=1 last;
rewrite ^/([^/]+?)-robots([0-9]+)?.txt$ /?robots=$1&robots_n=$2 last;
# END Nginx Rewrites for Rank Math Robots.txt File

    Please get in touch with your web host for assistance to insert it into your server config properly.

    Once done, clear your website and server’s cache and check again the robots.txt URL.

    Let us know how this goes.

    RankBot

    2 months, 1 week ago

    Hello,

    Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

    If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

    Thank you.

Viewing 2 replies - 1 through 2 (of 2 total)

The ticket ‘robots.txt throwing 404 error and Canonical URL is also missing’ is closed to new replies.