Generated Robots.txt file blocking googlebot

#149079
Viewing 6 replies - 1 through 6 (of 6 total)
  • Jeremy

    3 years, 2 months ago

    Hello,

    Thank you for contacting the support, and sorry for any inconvenience that might have been caused due to that.

    I checked your website’s robots.txt to a robots.txt validator and it seems that your robots.txt is correctly configured. Please see the screenshot below:
    robots.txt validator

    Can you please clear your website cache including hosted caching software like CloudFlare and resubmit your website to Google Search Console? See if the issue goes away after doing so.

    Let me know how it goes. We are looking forward to helping you.

    Carl Glancey

    3 years, 2 months ago

    Rank Math free

    After going back and forth with my hosting, they have cleared all caching on there end. I’ve resubmitted the pages in google search console and I still get the error that Googlebot is being blocked by robots.txt when looking at embedded google maps on my page.

    Is there anything else you can suggest?

    Jeremy

    3 years, 2 months ago

    Hello,

    Upon re-checking your robots.txt, I noticed that it is now loading an empty page.

    Can you please head over to your website’s root directory and see if you can find any static robots.txt file? If you find one, please delete it.

    After doing so, clear your cache once again and Rank Math should virtually generate a standard robots.txt file for you.

    If the issue persists, we might need to take a closer look at the settings. Please edit the first post on this ticket and include your WordPress & FTP logins in the designated Sensitive Data section.
    Sensitive Data Section

    It is completely secure and only our support staff has access to that section. If you want, you can use the below plugin to generate a temporary login URL to your website and share that with us instead:

    https://wordpress.org/plugins/temporary-login-without-password/

    You can use the above plugin in conjunction with the WP Security Audit Log to monitor what changes our staff might make on your website (if any):

    https://wordpress.org/plugins/wp-security-audit-log/

    We really look forward to helping you.

    Carl Glancey

    3 years, 2 months ago

    Rank Math free

    Hello,

    I have updated the sensitive data as requested. Can you please check further?

    Thank you.

    Carl Glancey

    3 years, 2 months ago

    Rank Math free

    The only thing that has changed from the robots.txt file not being blank and now appearing blank is the clearing of cache on the hosting side, nothing has been changed on the wordpress side so not sure why that robots.txt file is appearing blank now

    Reinelle

    3 years, 2 months ago

    Hello,

    Thank you for that information.

    Rank Math’s unable to generate the default robots.txt because there is an existing empty robots.txt file in your website’s root directory. However, I’ve already deleted that, and you can now access your robots.txt.

    This should fix the issue now. Could you please check on your end as well?

    Thank you, and looking forward to your update.

    RankBot

    3 years, 1 month ago

    Hello,

    Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

    If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

    Thank you.

Viewing 6 replies - 1 through 6 (of 6 total)

The ticket ‘Generated Robots.txt file blocking googlebot’ is closed to new replies.