How to avoid GSC to crawl search request

#479612
  • Resolved Irene RF

    2 years, 1 month ago

    Rank Math free

    Hi,

    I have a GSC crawling issue that is becoming bigger.

    1) Google crawled 249 URLs from spammy links to my site that begin with example.com/?s= going to site search box results (rankmath set to noindex) that get 404 error. They are considered crawled not indexed. Considering that my site has around 350 posts, this is a lot of crawling budget spent without purpose.
    How can I avoid Google crawling these links?

    2) Google was crawling Ezoic CDN with more than 2500 files instead of my origin site, I disavowed Ezoic directories in robot.txt. Is this the right thing to do?

Viewing 3 replies - 1 through 3 (of 3 total)
  • Nigel

    2 years, 1 month ago

    Rank Math business

    Hello,

    Thank you for contacting Rank Math for help with your search and crawling questions.

    1) Google crawled 249 URLs from spammy links to my site that begin with example.com/?s=…
    …How can I avoid Google crawling these links?

    You can disallow search URLs in your robots.txt file. To do that, add the following line to your robots.txt:
    Disallow: /*?s=*

    Google was crawling Ezoic CDN with more than 2500 files instead of my origin site, I disavowed Ezoic directories in robot.txt. Is this the right thing to do?

    If you disallowed crawling CDN URLs, they should eventually be removed from the index. Please wait for Google to attempt to recrawl the URLs which may take up to a few weeks.

    Hope that helps. Please let us know if you have questions.

    Irene RF

    2 years, 1 month ago

    Rank Math free

    Thanks I followed your advice.

    Reinelle

    2 years, 1 month ago

    Hello,

    Glad that helped.

    Please feel free to reach out to us again in case you need any other assistance.

    We are here to help.

    Thank you.

Viewing 3 replies - 1 through 3 (of 3 total)

The ticket ‘How to avoid GSC to crawl search request’ is closed to new replies.