AI Bot Crawler denied with RankMaths standard robots.txt feature

#1113043
  • Resolved Sven Oliver Heck
    Rank Math pro
    Content AI Starter Trial

    Hi, just reaching out as I was debugging AI Bots not being able to fetch my site.

    Even that your robots.txt checker said – everything fine – I did not get it to work only when I installed a real robots.txt file on my server. Now it works.

    I don’t know why I needed to put the robots.txt as an actual file and your checker said – all well. This tool afterwards confirmed accessibility. https://mrs.digital/tools/ai-crawler-access-checker/

    As this of course is somewhat problematic I thought I reach out so maybe that’s a bug.

Viewing 5 replies - 1 through 5 (of 5 total)
  • Hello,

    We apologize for the issue you are facing with the AI bots not being able to fetch your site. Thank you for raising this concern.

    It seems like you encountered a situation where the Rank Math robots.txt feature did not reflect the actual accessibility of your site until you uploaded a physical robots.txt file to your server. This can happen if there are discrepancies between the virtual settings in the plugin and the actual file on the server.

    To ensure that your robots.txt file is correctly configured, you can use our Robots Testing Tool to simulate how different user agents interact with your site. If you find that everything is working fine with the actual file, it may indicate a potential issue with how the plugin’s settings are being applied.

    If you continue to experience problems or believe this might be a bug, please reply to this ticket with any additional details, and a human agent from our support team will assist you further.

    Thank you for your understanding, and we appreciate your feedback as it helps us improve our services!

    Sources:
    Common robots.txt Issues & How to Fix Them » Rank Math
    How to Fix the "Indexed, though blocked by robots.txt" Error » Rank Math
    How to Block Unwanted Web Crawlers from Your Site » Rank Math

    Sven Oliver Heck
    Rank Math pro
    Content AI Starter Trial

    I also recognized that Claude is not able to “fetch” the page but can “search” the page – so there seems also to be different modes how AI Tools work with pages – but as long as my site is searcharble I am fine.

    Hello,

    Thank you for the update and for sharing your findings.

    Rank Math’s virtual robots.txt is fully compliant with standard protocols and works reliably for most major crawlers, including Googlebot and Bingbot. However, some AI bots or third-party tools may have limitations in how they interpret virtual files, especially if they’re expecting a physical robots.txt at the server level.

    That said, it’s not an issue with Rank Math itself, but rather how certain crawlers interact with WordPress’s virtual files. In such rare cases, adding a physical robots.txt can act as a fallback — as you’ve successfully done — though for most users, Rank Math’s built-in solution is more than sufficient.

    We’ll keep monitoring such edge cases to ensure compatibility where possible. If you run into anything unusual again, feel free to share details — your feedback helps us continually improve.

    Thank you.

    Sven Oliver Heck
    Rank Math pro
    Content AI Starter Trial

    Thanks for clarification

    Hello,

    We are glad we could address your concern.

    Please feel free to create a new forum topic if you do have another question in the future.

    It will be our pleasure to assist you again.

    Thank you.

Viewing 5 replies - 1 through 5 (of 5 total)

The ticket ‘AI Bot Crawler denied with RankMaths standard robots.txt feature’ is closed to new replies.