Ahrefs Bot Unable to access website on a server level

#232385
  • Resolved Rylan Agera
    Rank Math free

    I have added the message which Ahrefs customer representative sent me in Sensitive Data. They are saying that they can’t crawl properly because of some robot.txt problems. Could you please check this error as Rankmath handles robot.txt?

Viewing 7 replies - 1 through 7 (of 7 total)
  • Hello,

    Thank you for contacting the support, and sorry for any inconvenience that might have been caused due to that.

    Can you please share the affected URL with us so we can further check it on our end?

    We are looking forward to helping you.

    Hello,

    I have updated the sensitive data as requested. Can you please check further?

    Thank you.

    Hello,

    Thank you for providing the URL. I tried checking your robots.txt file and it seems to be in the correct configuration that allows bots to access it.

    I also tested it with robots.txt validator and it is working fine. Please refer to this screenshot: https://i.rankmath.com/JtOfIZ

    Could you please clear your website cache and let Ahrefs crawl your site again?

    Let us know how it goes. We are looking forward to helping you.

    Hey I contacted Ahrefs and it said that maybe my plugin sucuri is blocking it. Not sure but that’s the assumption.

    Any Thoughts? What should I do?

    Prabhat
    Rank Math agency

    Hello,

    You can reach out to the support team of Sucuri as they would be in a better position to assist you in this regard.

    If that doesn’t help, then as suggested by Ahrefs, please contact your hosting provider for further assistance.

    Hope that helps.

    Thank you.

    Hey
    Thanks for your help!

    Ankit
    Rank Math business

    Hello,

    Glad that helped.

    Please feel free to reach out to us again in case you need any other assistance.

    We are here to help.

    Thank you.

Viewing 7 replies - 1 through 7 (of 7 total)

The ticket ‘Ahrefs Bot Unable to access website on a server level’ is closed to new replies.