Hello,
I’m sorry to hear that you are facing issues with your robots.txt file. It seems like you have added the necessary rules to your robots.txt file, including the rule to disallow /cdn-cgi/.
If you are still encountering problems with your robots.txt file not being found, I recommend checking the following:
- Make sure that the robots.txt file is accessible at the root of your domain, for example, `https://technicalseo.com/tools/robots-txt/.
- Verify that there are no caching issues preventing the updated robots.txt file from being fetched by testing it in an incognito window or clearing your website cache.
- Double-check the permissions of your robots.txt file to ensure it is readable by search engine crawlers.
If after checking these points, the issue persists, please reply to this ticket, and a human agent from the Rank Math Support team will assist you further.
Thank you for your understanding and patience.
Sources:
Common robots.txt Issues & How to Fix Them » Rank Math
What Are the Best Robots.txt File Rules for Multisite? » Rank Math
Hello,
Thank you for contacting us and sorry for any inconvenience that this issue might have caused.
Visiting your site’s /robots.txt we can see there’s an error appearing ERR_TOO_MANY_REDIRECTS
Please note that the robots.txt file will not work if you have a robots.txt file present in your root folder. If you’ve any, you’ll need to delete it for Rank Math to generate a virtual robots.txt. Access your site through FTP, and delete the robots.txt file from there.
Once done, Rank Math should generate a dynamic robots.txt file for you.
Let us know how this goes. Also, let us know if you have further questions or concerns.
Thank you.
I followed the guidelines on how to access the site via SSH and SFTP on Cloudways and was unable to do either. I’ve never run into this issue before on my other sites with the same setup process and tech stack. Is there another way?
UPDATE: I was able to get in with some support, but there is no robots.txt file. I had added one to Rank Math but it was still telling me there was no robots.txt file. I’m unsure where to go from here.
Hello,
In that case, the issue seems to be occurring because your website tries to add a trailing slash to all your URLs. It seems it is trying to add it to the robots.txt also.
Please review that server rule and make the adjustments necessary so that it doesn’t add the slash when the URL being loaded has a file extension like the robots.txt file.
You may also ask your hosting provider to do that for you.
Let us know how it goes. Looking forward to helping you.
Thank you.
It was a caching issue through Cloudflare. I see the file now, but I added it manual through the SFTP since Rank Math did not.
Hello,
We are glad you could add the robots.txt file using SFTP. Rank Math should have generated a dynamic robots.txt as it does for many other sites and users.
It is fine to have a static robots.txt as one of the advantages of a dynamic robots.txt by Rank Math is that you can edit the robots.txt file from within Rank Math. In your case, you can edit the file from your SFTP, if you want to add a new rule and it should work correctly.
Hope that helps, and please do not hesitate to let us know if you need our assistance with anything else.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.
