Robot.txt file not updating, Google not able to index site.

#546317
  • Resolved Nazedesigns

    2 years, 9 months ago

    Rank Math free

    My client’s site was recently incorrectly flagged as having a phishing scam and was removed from google. I resolved the issue and Google put the site online again, however now I’ve requested it to be indexed it keeps coming up with “Failed: Blocked due to other 4xx issue”

    I was told by my host that my robot.txt file was only allowing my ajax file to be viewed, I thought I had since resolved this and replaced the code on the RankMath settings page, but the site is still bouncing back from being indexed with the same failed error.
    Does it take a few hours for the robot.txt changes to go through? Is there any advice from anyone to help resolve the issue?
    Many thanks in advance 😊

Viewing 11 replies - 1 through 11 (of 11 total)
  • Sandeepan Kumar

    2 years, 8 months ago

    Rank Math business

    Hello,

    Thank you for contacting the support and sorry for any inconvenience that might have been caused due to that.

    Would you please share the affected URL? Also, I would suggest to go through this article to understand if anything among these is causing the issue:
    https://rankmath.com/kb/blocked-due-to-other-4xx-issue/

    Let us know how it goes.

    Thank you.

    Nazedesigns

    2 years, 8 months ago

    Rank Math free

    Hey,

    The affected URL is: https://christinamcnicklecoach.com – and yes I’ve gone through that article already and haven’t had any luck 🙁

    Reinelle

    2 years, 8 months ago

    Hello,

    I’ve checked your robots.txt, and there is no directive/rule for disallowing your site to be crawled. Also, the line Allow: /wp-admin/admin-ajax.php does not mean that is the only file that crawlers/bots can crawl.

    It is considered a whitelist for the rule Disallow: /wp-admin/.

    Here are our recommended robots.txt rules for indexing:
    https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/#default-rules

    Hope that helps.

    Thank you.

    Nazedesigns

    2 years, 8 months ago

    Rank Math free

    Thank you for checking that. Why else might I be getting this error response and the crawl is bouncing back?

    Sandeepan Kumar

    2 years, 8 months ago

    Rank Math business

    Hello,

    Are you getting 4xx errors on any particular URLs? If yes, then please share few URLs so that we can look into the possible cause. Normally, 4xx issue is caused due to server misconfiguration and the hosting provider can resolve this issue as they have root access to their server.

    Please share some URLs which are affected by 4xx URLs.

    Looking forward to helping you.

    Thank you

    Nazedesigns

    2 years, 8 months ago

    Rank Math free

    Yes all URLS on the site are currently effected;
    https://christinamcnicklecoach.com
    https://Christinamcnicklecoach.com/contact

    I have checked with the host (siteground) and they referred me on to seek specific SEO help.

    Md. Sakib Khandaker

    2 years, 8 months ago

    Rank Math agency

    Hello,

    It looks like there’s an issue with country blocking for your URLs, since Google is throwing Teapot page while visiting your site. As the status code is 418, so you’re encountering 4xx error for your URLs.

    Here you can check:
    https://search.google.com/test/rich-results/result?id=C7-9sc3HlKoBFd1lTCALXw

    Please get in touch with your hosting provider in order to fix this one.

    Let us know how it goes. Looking forward to helping you.

    Thank you.

    Nazedesigns

    2 years, 8 months ago

    Rank Math free

    Ah I see, I had several countries blocked as there were a lot of attempted hacks. I’ve now deleted them, should I give it a few days before I request another crawl?

    Sandeepan Kumar

    2 years, 8 months ago

    Rank Math business

    Hello,

    Now, it looks fine as it passed the Google Rich result test.
    Though you can request recrawl now as well but I would suggest wait for few days and maybe Google will attempt recrawl by itself.

    Hope that helps and please do not hesitate to let us know if you need our assistance with anything else.

    Thank you

    Nazedesigns

    2 years, 8 months ago

    Rank Math free

    This solved the issue! It’s already showing up in the search results. Thanks everyone for your advice and help.

    Princess

    2 years, 8 months ago

    Hello,

    We are super happy that this resolved your issue. If you have any other questions in the future, know that we are here to help you.

    If you don’t mind me asking, could you please leave us a review (if you haven’t already) on https://wordpress.org/support/plugin/seo-by-rank-math/reviews/#new-post about your overall experience with Rank Math? We appreciate your time and patience.

    If you do have another question in the future, please feel free to create a new forum topic, and it will be our pleasure to assist you again.

    Thank you.

Viewing 11 replies - 1 through 11 (of 11 total)

The ticket ‘Robot.txt file not updating, Google not able to index site.’ is closed to new replies.