Robots.txt 404

#138642
  • Resolved Support Email
    free

    Hello,

    the virtual robots.txt that RankMath should generate gives a 404 back when requesting.
    The 404 Monitor that i’ve enabled doesn’t log anything either, so i’m out of ideas and need a little help here.

    Best Regards,
    JT

Viewing 8 replies - 1 through 8 (of 8 total)
  • Hello,

    Thank you for contacting Rank Math and sorry for any inconvenience that might have been caused due to that.

    Since your are using Nginx, please make sure you have added the following rules to your Nginx config file:

    
    # START Nginx Rewrites for Rank Math Sitemaps
    rewrite ^/sitemap_index.xml$ /index.php?sitemap=1 last;
    rewrite ^/([^/]+?)-sitemap([0-9]+)?.xml$ /index.php?sitemap=$1&sitemap_n=$2 last;
    # END Nginx Rewrites for Rank Math Sitemaps
    

    Let us know if this helps

    Looking forward to helping you. Thank you.

    ​​​​​​​

    Hello & thank you for the fast respond,

    i’ve already added the above lines in the nginx website configuration file as advised during the plugin installation und the Sitemap Settings menu.
    The sitemap itself works without any issue: https://redmonster.gr/sitemap_index.xml
    It’s just the robots.txt that gives a 404 back

    Regards
    JT

    Hello,

    Thank you for keeping in touch with us.

    Can you please check if you have a static robots.txt present in your root folder? Please delete if it does.

    Also, ensure that you are using the latest version of Rank Math, you may check our
    changelog here: https://rankmath.com/changelog/

    If the issue persists and the 404 logs are still not capturing 404 pages, we might need to take a closer look at the settings. Please edit the first post on this ticket and include your WordPress & FTP logins in the designated Sensitive Data section.
    Sensitive Data Section

    It is completely secure and only our support staff has access to that section. If you want, you can use the below plugin to generate a temporary login URL to your website and share that with us instead:

    https://wordpress.org/plugins/temporary-login-without-password/

    You can use the above plugin in conjunction with the WP Security Audit Log to monitor what changes our staff might make on your website (if any):

    https://wordpress.org/plugins/wp-security-audit-log/

    We really look forward to helping you.

    Hello,

    I have updated the sensitive data as requested. Can you please check further?

    Thank you.

    Hi,

    the production server serves other sites as well and i’m not allowed by the site owners to give ssh access to the server.
    Therefore i have mirrored the production site to the staging site server https://redmonster.ga and created a temporary admin login for you as advised above.
    The staging site is exact the same nginx and wordpress server configuration as the production site and has the same issue with the robots.txt as well, so you can examine the issue as needed.

    Thanks,
    JT

    Hello,

    It looks to be a common issue with WordPress on NGINX.

    I found this link from the WordPress.org website regarding it:
    https://wordpress.org/support/article/nginx/#global-restrictions-file

    You would need to use the code:

    
    location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
    }
    

    If this does not work please contact your hosting provider so they could provide you the correct rules.

    I hope this helps you with your issue.

    Hello Danial,

    i already have the above location ruleset in the config file.
    I’ll give it a try with the hoster.

    Thanks & Regards,
    JT

    Hello,

    I hope that will help you. Please feel free to reach out to us again in case you need any other assistance.

    Looking forward to helping you.

    Thank you.

Viewing 8 replies - 1 through 8 (of 8 total)

You must be logged in to reply to this ticket.