“robots.txt” – Conflicting Test Results – Lighthouse vs RankMath

#1020977
  • Resolved AJ

    5 months, 1 week ago

    Rank Math free

    Hello RankMath friends,

    I am a WP user with basic tech knowledge (using WP and learning as I come across issues) and really appreciate how much help your awesome plugin provides people like me.

    I am busy with creating a new WP site and am now in unfamiliar territory concerning robots.txt issues. The PageSpeed Insights test results state that “robots.txt is not valid – Lighthouse was unable to download a robots.txt file”

    So after doing a lot of reading on Astra (my theme) knowledge pages and then also your helpful knowledge pages about robots.txt I used your robots.txt test page which gave results stating that there are no issues with the robots.txt?

    I am very confused why Google says the robots.txt is not valid and there is no robots.txt file for it to download but RankMath robots.txt test results say everything is fine?

    Please can you help me with whatever the issue is because I spent a lot of time using your plugin to do all the SEO work on all the website pages and it will be a waste if Google test is complaining that robots.txt is not valid even if RankMath test says everything is fine.

    I have uploaded screenshots to Imgur as requested in your contact form.

    Thank you kindly!
    Arryn

Viewing 5 replies - 1 through 5 (of 5 total)
  • Rakibuzzaman Siam

    5 months, 1 week ago

    Hello,

    Thank you for contacting us, and sorry for any inconvenience that might have been caused due to that.

    This error is showing because Rank Math generates the robots.txt file dynamically, and there is no physical file present in your site’s root directory to download.

    In order to fix that error, you can follow the steps mentioned in this guideline: https://rankmath.com/kb/fix-common-robots-txt-issues/#robots-txt-is-not-valid-in-pagespeed-insights

    Hope that helps and please don’t hesitate to let us know if you have any other questions.

    AJ

    5 months ago

    Rank Math free

    Hello? I have not received any reply to my message a week ago? Please advise. Thank you kindly

    Reinelle

    5 months ago

    Hello,

    Apologies for the delay, as this ticket seems to have been stuck in our drafts section for some reason.

    The error is due to missing a physical robots.txt file, but it does not prevent any bots/crawlers from seeing the rules inside your https://yourdomain.com/robots.txt.

    If you want to fix it, you can create a static robots.txt directly in your website’s root directory and copy our recommended rules.

    However, you won’t be able to update it in Rank Math > General Settings since there’s already a file:
    https://rankmath.com/kb/cant-edit-robots-txt/

    Looking forward to helping you.

    AJ

    5 months ago

    Rank Math free

    Hello Reinelle,

    Thank you for your kind reply.

    Maybe I am not asking my question correctly for the answer I am looking for…

    I understand now that RankMath robots.txt is dynamic and not actually a physical file – thank you.

    Google’s PageSpeed Insights test results state that (the RankMath) “robots.txt is not valid – Lighthouse was unable to download a robots.txt file”

    So please can you tell me: Does this negatively affect my website with Google?

    Thank you kindly 🙂

    Rakibuzzaman Siam

    5 months ago

    Hello,

    No, this is not an issue which will negatively affect your SEO. In this test, Google only checks if Lighthouse can download the robots.txt file or not.

    We suggest you submit your robots.txt on your GSC account, and if Google can fetch that properly, you don’t need to worry about this.

    Hope that helps and please don’t hesitate to let us know if you have any other questions.
     
    Thank you.

    RankBot

    4 months ago

    Hello,

    Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

    If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

    Thank you.

Viewing 5 replies - 1 through 5 (of 5 total)

The ticket ‘“robots.txt” – Conflicting Test Results – Lighthouse vs RankMath’ is closed to new replies.