Edit.robots.txt. file

#1010344
  • Resolved Navjyot Gill
    Rank Math free

    Ahrefs site audit detected that my website could not be crawled and pointed to the Edit.Txt file. On checking in Rank Math, I noticed that 1] robot.txt is not writable in red, and 2] the sitemap URL created by Rank Math is erroneous [404 error]. I have created a redirection to the correct sitemap URL, but how do I change this in Rank Math to prevent it from happening again? I have suffered a massive drop in page views and traffic over the last 15 days.
    Also, why did Rank Math automatically create this?
    Please help me to rectify these errors immediately.
    Thanks.

Viewing 7 replies - 1 through 7 (of 7 total)
  • Hello,

    We apologize for the issues you are facing with your website’s robots.txt file and sitemap URL. Thank you for bringing this to our attention.

    Addressing the Robots.txt File Not Writable Issue

    The error indicating that the robots.txt file is not writable typically occurs when Rank Math cannot modify your robots.txt file. To resolve this, you can add the following code to your site:

    /**
     * Allow editing the robots.txt & htaccess data.
     *
     * @param bool Can edit the robots & htacess data.
     */
    add_filter( 'rank_math/can_edit_file', '__return_true' );

    Additionally, check your wp-config.php file for the following line:

    define (‘DISALLOW_FILE_EDIT’, true);

    If it exists, change it to:

    define ('DISALLOW_FILE_EDIT', false);

    If it does not exist, you can add it. If you are unable to access your wp-config.php file, please contact your hosting provider for assistance.

    Fixing the 404 Error for the Sitemap URL

    To address the 404 error for the sitemap URL, ensure that there are no conflicting plugins or physical files that might be causing this issue. You can check if another plugin is creating a sitemap or if a physical sitemap file exists in your site’s root directory. If so, you may need to disable that plugin or remove the physical file.

    After making these changes, clear your website and server-level cache and test your sitemap URL again. You can refer to this guide on how to clear cache in WordPress.

    Why Did Rank Math Automatically Create This?

    Rank Math automatically creates a virtual robots.txt file to help manage SEO settings directly from the WordPress dashboard. This means you do not need a physical robots.txt file in your website’s root folder. If a physical file exists, it can conflict with Rank Math’s virtual file, leading to issues like the ones you’re experiencing.

    Next Steps

    Please reply to this ticket with your site URL so we can assist you further if needed. A human agent from our support team will take over and provide additional help if necessary.

    We appreciate your patience as we work to resolve these issues for you!

    Sources:
    I Can’t Edit My Website’s Robots.txt » Rank Math
    Common robots.txt Issues & How to Fix Them » Rank Math

    I am a novice and do not understand the robots.txt technicalities. I would appreciate it if Rank Math resolved this issue, as it was created by them in the first place.
    An early rectification will be appreciated.
    Warmly,
    Navjyot Gill

    Hello,

    In this case, we might need to take a closer look at the settings. Please edit the first post on this ticket and include your WordPress & FTP logins in the designated Sensitive Data section.

    Please do take a complete backup of your website before sharing the information with us.

    Sensitive Data Section

    It is completely secure and only our support staff has access to that section. If you want, you can use the below plugin to generate a temporary login URL to your website and share that with us instead:

    https://wordpress.org/plugins/temporary-login-without-password/

    You can use the above plugin in conjunction with the WP Security Audit Log to monitor what changes our staff might make on your website (if any):

    https://wordpress.org/plugins/wp-security-audit-log/

    We really look forward to helping you.

    I was hoping for a quick resolution to the issues I was facing. Despite having sent you all the details you requested, I have still not heard from you.
    I expect at least a reply about the current status.
    Thanks!

    Hello,

    We deeply apologize for the unexpected delay in response. This ticket has been stuck in our drafts section for some reason.

    However, we tried to check your websites, but the temporary login URL you shared has expired and is getting redirected to the login page.

    Please renew and extend the login to at least 3 days, and share this in the sensitive data section so we can check the issue further.

    Looking forward to helping you.

    Thank you.

    So, I finally get a response from you requesting to go through the whole process again.
    I could not wait indefinitely and hired someone to fix it.
    But this careless attitude was not expected of Rank Math.
    Probably, you prioritize Pro members?
    I am not marking this as resolved as you did not fix it.
    Hoping for better attention and service in the future.

    Hello,

    We sincerely apologize again for the inconvenience caused.

    We understand how frustrating the delay must have been and truly regret that we couldn’t assist you in time.

    Due to a recent surge in support requests, our response times have been slower than usual. We’re working hard to improve this and ensure timely support for all users.

    If there’s anything we can still help with, please don’t hesitate to reach out.

    We appreciate your patience and understanding.

    Hello,

    Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

    If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

    Thank you.

Viewing 7 replies - 1 through 7 (of 7 total)

The ticket ‘Edit.robots.txt. file’ is closed to new replies.