Hello,
We apologize for the issue you are facing with the AI bots not being able to fetch your site. Thank you for raising this concern.
It seems like you encountered a situation where the Rank Math robots.txt feature did not reflect the actual accessibility of your site until you uploaded a physical robots.txt file to your server. This can happen if there are discrepancies between the virtual settings in the plugin and the actual file on the server.
To ensure that your robots.txt file is correctly configured, you can use our Robots Testing Tool to simulate how different user agents interact with your site. If you find that everything is working fine with the actual file, it may indicate a potential issue with how the plugin’s settings are being applied.
If you continue to experience problems or believe this might be a bug, please reply to this ticket with any additional details, and a human agent from our support team will assist you further.
Thank you for your understanding, and we appreciate your feedback as it helps us improve our services!
Sources:
Common robots.txt Issues & How to Fix Them » Rank Math
How to Fix the "Indexed, though blocked by robots.txt" Error » Rank Math
How to Block Unwanted Web Crawlers from Your Site » Rank Math
I also recognized that Claude is not able to “fetch” the page but can “search” the page – so there seems also to be different modes how AI Tools work with pages – but as long as my site is searcharble I am fine.
Hello,
Thank you for the update and for sharing your findings.
Rank Math’s virtual robots.txt is fully compliant with standard protocols and works reliably for most major crawlers, including Googlebot and Bingbot. However, some AI bots or third-party tools may have limitations in how they interpret virtual files, especially if they’re expecting a physical robots.txt at the server level.
That said, it’s not an issue with Rank Math itself, but rather how certain crawlers interact with WordPress’s virtual files. In such rare cases, adding a physical robots.txt can act as a fallback — as you’ve successfully done — though for most users, Rank Math’s built-in solution is more than sufficient.
We’ll keep monitoring such edge cases to ensure compatibility where possible. If you run into anything unusual again, feel free to share details — your feedback helps us continually improve.
Thank you.
Hello,
We are glad we could address your concern.
Please feel free to create a new forum topic if you do have another question in the future.
It will be our pleasure to assist you again.
Thank you.
