website is not fetched by google crawler

#39779
  • Resolved Rajparitosh
    Rank Math free

    Google crawler is unable to index my website. The error message in Crawl allowed? section is “No: blocked by robots.txt”. In-Page fetch section also there is a red flag with error message”Failed: Blocked by robots.txt”. I checked the robots.txt there is no error.

    My whole website is showing error and is unreachable by google bot. However, it is accessible from web browser and is fetched by gtmatrix and Pingdom.
    The website is https:top10gears.com
    Previously it was indexed by google.

    • This topic was modified 4 years, 7 months ago by Rajparitosh.
Viewing 10 replies - 1 through 10 (of 10 total)
  • Hello,

    Thanks for getting in touch with us today.

    This could be related to the fact that your sitemap page https://top10gears.com/sitemap_index.xml is reporting a 404 error.

    Could you please do the following:

    1. Head over to WordPress Dashboard > Settings > Permalinks

    Hit the save button once.

    2. Go to WordPress Dashboard > Rank Math > Sitemap Settings

    Hit the save button once.

    That should fix the issue.

    If it doesn’t, please do let us know so we can look at this further.

    I followed all the above-said steps but after following the steps, I inspected the URL with URL Inspection in google search console and it shows the same error.

    hai, i also have the same issue after the update. please help

    Hello,

    We might need to take a closer look at the settings. Please edit the first post on this ticket and include your WP logins in the designated Sensitive Data section.
    Sensitive Data Section

    It is completely secure and only our support staff has access to that section. If you want, you can use the below plugin to generate a temporary login URL to your website and share that with us instead:

    https://wordpress.org/plugins/temporary-login-without-password/

    You can use the above plugin in conjunction with the WP Security Audit Log to monitor what changes our staff might make on your website (if any):

    https://wordpress.org/plugins/wp-security-audit-log/

    We really look forward to helping you.

    do I need to create a new ticket since I’m not the ticket owner? but I have similar problem

    Hi Ahmad,

    Thanks for the follow up.

    Yes you need to create a new ticket to access the sensitive data section.

    We are looking forward to helping you. Thank you.

    Hello,

    I have updated the sensitive data as requested. Can you please check further?

    Thank you.

    Hi Raj,

    Thanks for the follow up.

    I have checked your site settings and the noindex option is not enabled.

    Please clear your site cache and the validate this fix.

    Let me know if the issue is still reported.

    Thank you.

    Hi Michael,

    Sorry to inform you but the problem is still persistent.

    Todd
    Rank Math free

    Hi,

    I am sorry but I do not see any noindex tag or nothing of the sorts in the robots.txt file either.

    There is no noindex tag and nothing that would prevent your homepage from getting crawled. You can verify the same by testing on 3rd party tools as well.

    You have to contact Google and see what’s going on. As far as Rank Math is concerned – there is no issue.

    Hello,

    Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

    If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

    Thank you.

Viewing 10 replies - 1 through 10 (of 10 total)

The ticket ‘website is not fetched by google crawler’ is closed to new replies.