Page cannot be indexed: blocked by robots.txt

#472224
  • Resolved Emma Wienke
    Rank Math free

    Hi,

    I’ve launched a website and in Google Search Console it says that 10 out of 12 pages cannot be indexed because they were excluded by ‘noindex’ tag.

    I have followed all the instructions in your blog post https://rankmath.com/kb/excluded-by-noindex-tag/.
    In WordPress Dashboard > Settings > Reading > Search engine visibility the box is unticked.
    In Rank Math > general settings > edit robots.txt reads User-agent: *
    Disallow:
    In Titles and meta global meta > robots meta I have ticked to index and the same for posts and pages.

    Google still won’t index the website even when I do a fetch live page.

    I don’t know what else to do. Please can you help?

Viewing 5 replies - 1 through 5 (of 5 total)
  • Anas
    Rank Math business

    Hello,

    Thank you for contacting Rank Math, and sorry for any inconvenience caused.

    Can you please share the affected URLs with us so we can check?

    Also, please make sure that the pages are set to index from the Advanced Tab in the SEO Meta Box:
    metbox

    We look forward to helping you.

    Hello,

    I have updated the sensitive data as requested. Can you please check further?

    Thank you.

    Hello,

    I’ve checked your robots.txt, and it seems that you have a disallow rule set for your site.

    In this case, please update your robots.txt to our recommended rules here:
    https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/#default-rules

    Once done, give Google some time to recrawl your site and revalidate the issues on your Google Search Console account.

    Hope that helps.

    Thank you.

    I’ve done as you have advised and I will give Google some time to crawl the site. Thank you for your help!

    Hello,

    Glad that helped.

    Sure, let us know how this goes after giving Google some time to recrawl your site.

    Meanwhile, please don’t hesitate to contact us anytime if you have other questions to assist you further.

    Looking forward to helping you.

    Thank you.

Viewing 5 replies - 1 through 5 (of 5 total)

You must be logged in to reply to this ticket.