Lighthouse cannot download robots.txt file

#700755
  • Resolved Kenneth Gelinas

    4 months, 2 weeks ago

    Rank Math free

    Hi there, I just switched from AIOSEO to Rank Math and really like the detailed options and analysis for SEO scoring. I have a question though. I’ve run tests with google’s pageinsights and just started getting lower scores for their SEO portion. I’m talking about a drop from 100 to 92. It started yesterday, before that I was always seeing 100.

    My concern is that I’m continuing to see the same message that I received from the pageinsights test when I was using AIOSEO. That was one of the reasons for switching to Rank Math. I was unable to make changes to my robots.txt file with AIOSEO, so I switched to Rank Math hoping the problem would go away. But it hasn’t. I just ran the pageinsights test again after switching to Rank Math and I get the same error message that’s shown in the attached screenshot.

    Is there a problem with Lighthouse? It seems odd that I would get the same error message for both AIOSEO and Rank Math when the robots.txt files are different.

    I’m not sure what to do. I’m bringing this to your attention. Maybe google has made changes to their criteria and AIOSEO and Rank Math will have to make some adjustments in their software to accomodate for the changes.

    Please let me know if I could do anything to correct the issue.

    Thanks for your help. Ken

    This is the error message:

    robots.txt is not validLighthouse was unable to download a robots.txt file

Viewing 6 replies - 1 through 6 (of 6 total)
  • RankBot

    4 months, 2 weeks ago

    Hello,

    We apologize for the inconvenience you’re experiencing with the robots.txt file. The error message you’re seeing, “robots.txt is not valid – Lighthouse was unable to download a robots.txt file,” is a common issue when using a content delivery network (CDN) like Cloudflare.

    Rank Math does not create a physical robots.txt file on your site. Instead, the file is created and displayed when a user visits your robots.txt URL. So you can safely ignore this error if your robots.txt file is available at https://yourdomain.com/robots.txt.

    However, if you’d like to address this error, you can add the below rule to your robots.txt file and test it again in PageSpeed Insights:

    Disallow: /cdn-cgi/

    This should help resolve the issue you’re facing. If you have any further questions or need additional assistance, please feel free to ask.

    Thank you for bringing this to our attention.

    Sources:
    Common robots.txt Issues & How to Fix Them » Rank Math
    Rank Math Sitemap Issues and Their Fixes » Rank Math

    Reinelle

    4 months, 2 weeks ago

    Hello,

    If you are using our plugin to modify this file, the error is normal since we don’t create a physical file on your website but create a virtual one.

    This method is completely acceptable by Google, but if an application requests the actual file download, it will fail because the file doesn’t exist.

    ​​​​​​​To get around this issue, you need to create a physical file on your website, but then you would lose the option to edit it with Rank Math.

    You can copy the current rules in your robots.txt to the physical robots.txt file in your root directory.

    Hope that helps.

    Kenneth Gelinas

    4 months, 2 weeks ago

    Rank Math free

    Hi Reinelle, thanks for providing the additional code. I’m unable to add it to the existing code. I can’t add it to what’s already there. If I do it wipes out everything that was there before and only leaves the one line that you have provided me. Is there a way for you to send me the complete robots.txt file so that I can paste it and save it properly?

    Kenneth Gelinas

    4 months, 2 weeks ago

    Rank Math free

    Or send me the complete robots.txt file and I’ll add it to the public folder.

    Kenneth Gelinas

    4 months, 2 weeks ago

    Rank Math free

    Sorry I figured it out. Thanks again for your help!

    Adetayo

    4 months, 2 weeks ago

    Hello,

    We are glad to know that this issue has been resolved. Thank you for letting us know.

    This ticket will be closed now, but you can always open a new one if you have any other questions or concerns. We are here to help you with anything related to Rank Math.

    We appreciate your patience and cooperation throughout this process.

    Thank you for choosing Rank Math.

Viewing 6 replies - 1 through 6 (of 6 total)

The ticket ‘Lighthouse cannot download robots.txt file’ is closed to new replies.