robots blocking google but no disallow in robots

#433978
  • Resolved Easy Mode Media

    3 years, 2 months ago

    Rank Math free

    Hey everyone!
    Have an issue I can’t wrap my head around.
    Tried to crawl a homepage of mine and Google is saying the URL can’t be fetched due to being blocked by the robots.txt file, when I look at the robots.txt, I can’t see any reason for this being blocked.
    I read that plugins cause conflicts with the robots file, but I’m not sure any of these plugins would cause the issue. Any ideas of what I can try?
    Thanks!

Viewing 6 replies - 1 through 6 (of 6 total)
  • Anas

    3 years, 2 months ago

    Rank Math business

    Hello,

    Thank you for contacting Rank Math, and sorry for any inconvenience caused.

    I checked the URL on Google, it is already indexed and the robots.txt is not blocking the URLs.

    However, as the Live test is failing. Can you please clear the cache from your CDN and your website cache and live test the URL again?

    Let us know if the issue persists.

    We are here to assist you.

    Easy Mode Media

    3 years, 2 months ago

    Rank Math free

    Hi there, thanks for the response Anas,

    I’ve cleared my website cache (SiteGround) but I’m not sure the client has a CDN installed? I also tried a plugin cache to try and clear, but this hasn’t solved the issue.

    Easy Mode Media

    3 years, 2 months ago

    Rank Math free

    I’m not sure if it helps or not, but no other website crawlers have issues getting through to the site (SEMrush, Screaming Frog, etc).

    Anas

    3 years, 2 months ago

    Rank Math business

    Hello,

    I checked with the robots.txt tester, the URLs are not blocked by the file.

    Check the screenshot in the sensitive data section.

    Can you please check if there is any robots.txt file in the root directory of your website that is blocking any bots?

    If the robots.txt file is present in the root directory, please delete that file.

    You can also ask your web host to clear the CDN cache and see if that helps.

    Let us know if the issue persists.

    We are here to assist you.

    Easy Mode Media

    3 years, 2 months ago

    Rank Math free

    “Can you please check if there is any robots.txt file in the root directory of your website that is blocking any bots?”

    Where would I find this file? Not too familiar with the root directory area of WP.

    I contacted the webhost, there is no CDN active.

    Thankk you

    Jaideep Asrani

    3 years, 2 months ago

    Hello,

    You will find the file in the root directory where you see the folders like wp-content, wp-includes, etc. If you are not sure, we might need to take a closer look at the settings. Please edit the first post on this ticket and include your WordPress & FTP logins in the designated Sensitive Data section.

    Please do take a complete backup of your website before sharing the information with us.
    Sensitive Data Section

    It is completely secure and only our support staff has access to that section. If you want, you can use the below plugin to generate a temporary login URL to your website and share that with us instead:

    https://wordpress.org/plugins/temporary-login-without-password/

    You can use the above plugin in conjunction with the WP Security Audit Log to monitor what changes our staff might make on your website (if any):

    https://wordpress.org/plugins/wp-security-audit-log/

    We really look forward to helping you.

    RankBot

    3 years, 1 month ago

    Hello,

    Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

    If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

    Thank you.

Viewing 6 replies - 1 through 6 (of 6 total)

The ticket ‘robots blocking google but no disallow in robots’ is closed to new replies.