Robots txt timeout

Hello,

We apologize for the issues you are facing with your website’s robots.txt file and sitemap indexing.

1. Robots.txt Unavailable and Timing Out

If your robots.txt file is timing out, it could be due to server issues or misconfigurations. Rank Math creates a virtual robots.txt file, so you do not need a physical one in your website’s root folder. To troubleshoot this:

• Ensure that your server is running properly and can handle requests.
• Clear any caching plugins you might be using to ensure the latest version of the robots.txt file is being served.
• You can also check your robots.txt file using the Robots Testing Tool to see if it returns any errors.

2. Sitemap Issues in Google Search Console

For the sitemap issues, here are a few steps you can take:

• Validate Your Sitemap: Make sure your sitemap is correctly configured by entering your sitemap URL into the Validate XML Sitemap tool.
• Check Sitemap Location: Ensure that your sitemap is located at https://yourdomainname.com/sitemap_index.xml.
• Index Settings: Verify that your posts and pages are set to index. You can do this by going to Rank Math SEO → Titles & Meta and ensuring the Index checkbox is enabled for your posts and pages.

If these steps do not resolve the issues, please reply to this ticket, and a human agent from the Rank Math Support team will assist you further.

Thank you for bringing this to our attention!

Hello,

We checked your robots.txt file and we can see Googlebot can access it without any issues. Please check the screenshot in the sensitive data section for reference.

However, if Ahrefs is looking for a physical robots.txt file, it will not be able to find it as Rank Math generates the robots.txt file of your website on the fly.

Regarding the sitemap issue, we can see it is also working properly. In this case, please try to follow the steps below and see if that works for you:

1. Flush the Sitemap cache by following this video screencast:
https://i.rankmath.com/pipRDp

2. Exclude the Sitemap files of the Rank Math plugin in your caching plugin. The cache could be via a plugin or from the server. For plugins or Cloudflare, please follow this article:
https://rankmath.com/kb/exclude-sitemaps-from-caching/

Once done, clear your website cache, remove all the submitted sitemap, and resubmit only the /sitemap_index.xml in the Search Console.

If the issue persists, please try to follow this guideline for further troubleshooting: https://rankmath.com/kb/couldnt-fetch-error-google-search-console/

Let us know how it goes. Looking forward to helping you.

Thank you.