robots.txt gets a 404

#62046
  • Resolved Mark Constable
    Rank Math free

    Please excuse my confusion with my robots.txt generating a 404. From what I have read if I have no real robots.txt file in the root of my server then a virtual one should be generated but if I go to https://spiderweb.com.au/robots.txt I get a 404. Is there a module that needs to be enabled perhaps?

Viewing 13 replies - 1 through 13 (of 13 total)
  • Alberto
    Rank Math business

    Hello,

    Thank you for contacting Rank Math and sorry for any inconvenience that might have been caused due to that.

    We might need to take a closer look at the settings to try to determine the cause of the issue. Please edit the first post on this ticket and include your WP logins in the designated Sensitive Data section.
    Sensitive Data Section

    It is completely secure and only our support staff has access to that section. If you want, you can use the below plugin to generate a temporary login URL to your website and share that with us instead:

    https://wordpress.org/plugins/temporary-login-without-password/

    You can use the above plugin in conjunction with the WP Security Audit Log to monitor what changes our staff might make on your website (if any):

    https://wordpress.org/plugins/wp-security-audit-log/

    We really look forward to helping you.

    Hello,

    I have updated the sensitive data as requested. Can you please check further?

    Thank you.

    Hello,

    Thank you for contacting Rank Math and sorry for any inconvenience that might have been caused due to that.

    I have been able to replicate your issue. Allow me to consult with the dev team and get back to you.

    Looking forward to helping you. Thank you.

    ​​​​​​​

    Todd
    Rank Math free

    Hi Mark,

    There is something off with your website.

    I added a real robots.txt file to your root and it is still throwing a 404:

    Could you please contact your web host and see what is going on?

    I just installed Rank Math on this site and the same thing. Going to this URL just gets a 404 as well (this site is not on Cloudflare yet)…

    https://mandaleystud.com.au/robots.txt

    Is it possible that nginx needs a special rewrite rule?

    Hello,

    Thank you for contacting Rank Math and sorry for any inconvenience that might have been caused due to that.

    Please check your nginx config file and ensure there are no directives blocking robots.txt on your site.

    Looking forward to helping you. Thank you.

    ​​​​​​​

    The config can’t be blocking any robots.txt because if it really exists as a file in the root of the website then it works just fine.

    There is nginx advice for rewrite rules to show a virtual sitemaps_index.xml. Is it possible that something similar is needed to show a virtual robots.txt ?

    Hello,

    Thank you for the follow up.

    Allow me to consult on this and will get back to you.

    Looking forward to helping you. Thank you.

    Todd
    Rank Math free

    Hi,

    It is possible that your web server is not setup to handle .txt files. It is really best if you can contact your web host about this since they will be in a better position to assist.

    This is not something related to Rank Math as we have already established that by creating a robots.txt file and it doesn’t work in the browser.

    As I said previously, I am the hosting provider with full root access to the server. The nginx server can definitely handle txt files because when I put in a real robots.txt it works fine. It’s just the virtual one supposedly produced by Rant Math that is not showing up. This is a real robots.txt file…

    https://spiderweb.com.au/robots.txt

    Todd
    Rank Math free

    Hi Mark,

    That’s weird.

    Could you please share your FTP details with us as well so we can look at this in more detail?

    Thank you.

    Hi Todd, I’m a little wary of letting you guys in again. After someone got in and installed WP File Manager to check out the file system I had a hell of a time trying to log in myself to regain control of the website. I’m not sure if it was just a coincidental accident (or possibly malicious?) but I don’t want to go through that again. It took me over an hour with wp-cli to get back in. I was going to roll out your plugin to 100 sites over the next few weeks but between no virtual robots.txt and the above incident, I’ll reevaluate SEO plugins yet again.

    For now, I’ll just use a real robots.txt file.

    FWIW I removed some example nginx rewrite rules for sitemap_index.xml and it still shows up so something inside the plugin must provide the rewrite.

    Todd
    Rank Math free

    Hi Mark,

    We are sorry about that.

    Do let us know if you have any other queries you want to ask.

    We are only a ticket away.

Viewing 13 replies - 1 through 13 (of 13 total)

The ticket ‘robots.txt gets a 404’ is closed to new replies.