Robots.txt not responsive – Google not crawling actual pages

#246492
  • Resolved Hugo Alais

    4 years, 3 months ago

    Rank Math free

    Hi there,

    So I have this problem where Google is crawling all the wp content on my site but not the actual pages I publish.

    I added in these commands to robots.txt

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/plugins/
    Disallow: /wp-content/themes/
    Disallow: /wp-content/plugins/elementor

    But still Google crawls wp content and not the pages I’m actually publishing. So I was wondering if it might be a problem with Rank Math?

    Kind regards,
    Hugo

Viewing 6 replies - 1 through 6 (of 6 total)
  • Reinelle

    4 years, 3 months ago

    Hello,

    Thank you for contacting Rank Math and bringing your concern to our attention. I’m sorry for any inconvenience this issue may have caused you.

    Could you please provide your website URL to check further?

    Also, please check your Google Seach Console account if you have any reported coverage issue on your site
    Coverage

    Looking forward to helping you.

    Thank you.

    Hugo Alais

    4 years, 3 months ago

    Rank Math free

    Hello,

    I have updated the sensitive data as requested. Can you please check further?

    Thank you.

    Hugo Alais

    4 years, 3 months ago

    Rank Math free

    Hi Reinelle,

    I added the site in the sensitive data area.

    Reinelle

    4 years, 3 months ago

    Hello,

    I’ve checked your site, but I’m unable to replicate the issue on the search results pages. Please see the screencast I’ve attached in the sensitive data section.

    Could you please share the wp-content URL you are seeing to check and verify the issue here?

    Looking forward to helping you.

    Thank you.

    Hugo Alais

    4 years, 3 months ago

    Rank Math free

    Hi Reinelle,

    Those were pages that were crawled in the past. For the last three weeks no new pages I publish are getting crawled unfortunetly.

    Ankit

    4 years, 3 months ago

    Rank Math business

    Hello,

    There is one recently indexed page, As you can see in the screenshot:
    https://i.rankmath.com/s2Q4XD

    Search Engines are free to index websites and their contents. Some posts might take longer than others to get indexed. As there is no Hard and Fast rule for appearing on Google, the only solution is to optimize your content to increase their chances of appearing on Google. You should consider optimizing your content and leave the rest on Google.

    For more details, you can have a look at this article: https://www.socmedsean.com/answered-how-long-does-it-take-for-my-blog-posts-to-rank-in-google/

    We have a great tutorial that you might find useful in optimizing your contents: https://rankmath.com/kb/score-100-in-tests/

    As of now, you can keep an eye on the GSC coverage section for any possible errors and warnings and fix that accordingly.

    I hope that helps. Thank you, and please don’t hesitate to contact us anytime if you need further assistance.

    RankBot

    4 years, 2 months ago

    Hello,

    Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

    If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

    Thank you.

Viewing 6 replies - 1 through 6 (of 6 total)

The ticket ‘Robots.txt not responsive – Google not crawling actual pages’ is closed to new replies.